Bulk Review of Multiple-pieces Contents
Introduces how to call the interface to perform bulk review of multiple contents in the WebService application channel.
Bulk Review
Introduces calling the WebService application channel to send a JSON text via
to perform bulk review of contents. The Content-type in the HTTP header needs to be set to "application/json".
POST
This article uses batch review of Swift content as an example.
Request method
POST
URL
Request URL
/skg/v1/dlp/channel/webserviceapp/<WebService Application ID>/bulk_async
Request Parameters
| Parameter Name | Type/Required | Description |
|---|---|---|
| callback_url | String (Required) | The URL for asynchronous callback, which will be called after all objects have been checked. |
| verbose | Bool (Required) | Whether to return incident_info. |
| objects | List (Required) | A list of objects to be checked, with specific parameters referred to in the following table. |
The following table provides a detailed introduction to the specific parameters of the objects list.
| Parameter Name | Type | Description |
|---|---|---|
| user | String (Required) | The username that generates the event - supports domain users, in the format domain\username. |
| filename | String (Optional) | The file name - can be used for policy matching based on the file name. |
| customAttribute | String (Optional) | Supports user-defined parameter names. |
| queryID | String (Required) | The event query ID associated with this request, must be unique. If the request does not generate an event, the event details cannot be queried. Note: The value of queryid corresponds to the traffic UUID in the third-party cloud service. |
| redaction | Dict (Optional) | Related to redaction functionality. Whether to enable redaction. For specific redaction settings, please refer to the sendBack parameter. |
| sendBack | Dict (Optional) | Related to redaction functionality. Handling of redacted content. Note: To use the redaction feature, set the value of the redaction parameter to true. This field requires an input of a required type attribute, indicating the way the file is returned. The Type attribute supports the following options: response: Return the redacted content to the current path httpUpload: Send the redacted content to the specified URL s3: Upload the redacted content to Amazon S3 storage Note: This step also applies to storage in Swift, COS, OSS, etc. For specific related configuration parameters, please refer to the section objectinfo Parameters for Reviewing Cloud Service Stored Content in Asynchronous Mode.
|
| md5 | String (Optional) | The MD5 value of the file, used for recording and cache acceleration. Supports sending MD5 together with the document for inspection, as well as sending only MD5 for inspection. |
| operation | int (Optional) | Operation ID. Defined when creating the WebService application, and the operation name corresponding to the operation ID will be displayed in the Webservice Operation field of the API traffic log. There are four defaults, and users can also define their own:
|
| callback_url | String (Optional) | The listening address for callback |
| verbose | String (Required) | Whether to return event information. |
| uploadtype | String (Required) | Only for asynchronous. Supports AWS S3, Alibaba Cloud OSS, Tencent Cloud COS, Swift, as well as local files and download URLs.
|
| X-Auth-User | String (Required) | The authentication username for Swift. |
| X-Auth-Key | String (Required) | The authentication Key for Swift. |
| X-Auth-Token | String (Required) | The authentication token for Swift. |
| objectUrl | String (Required) | The address of the authenticated file for Swift. |
Request Example
The following example shows how to call the interface to batch upload files in the WebService channel for content security review.
{
"callback_url": "http://172.22.113.49:5000/post.php ",
"verbose" : True, // Whether to return incident_info
"objects": [
{
"user": "abc\enduser1",
"filename": "confidential.doc",
"queryID": "cd2fd109-c4d4-489f-9b27-53752f7827d6",
"uploadtype": "swift",
"objectInfo" : {
"authUrl" : "http://swift.example.com/auth/v1.0 ",
"X-Auth-User" : "testuser",
"X-Auth-Key" : "2121212",
"X-Auth-Token" : "AUTH_tkbdac2a3474ee4d6396133c99cfd962c",
"objectUrl" : "https://swift-server/confidential.doc"
}
},
{
"user": "abc\enduser2",
"filename": "test.doc",
"queryID": "47992d42-04b7-4860-b186-b8c11f8b2253",
"uploadtype": "swift",
"objectInfo" : {
"authUrl" : "http://swift.example.com/auth/v1.0 ",
"X-Auth-User" : "testuser",
"X-Auth-Key" : "2121212",
"X-Auth-Token" : "AUTH_tkbdac2a3474ee4d6396133c99cfd962c",
"objectUrl" : "https://swift-server/confidential.doc"
}
}
]
}
Python Request Example - Batch Inspection
The following example shows how to use Python code to call the interface to batch upload files in the WebService channel for content security review.
import json
import requests
auth_headers = {
"Authorization": "<Authorization>",
"x-skg-timestamp": "<timestamp>",
}
ucwi_endpoint = "<ENDPOINT>"
url = ucwi_endpoint + "/skg/v1/dlp/channel/webserviceapp/78217a43-542d-47e2-97a5-4435bf11521d/bulk_async"
data = {
"callback_url": CALLBACK_URL,
"verbose": False,
"objects": [
{
"user": "cloudtest\\clouduser",
"filename": "file1",
"queryID": "<queryID1>",
"uploadtype": "swift",
"objectInfo": {
"authUrl": "http://172.22.117.120:8080/auth/v1.0 ",
"X-Auth-User": "admin:admin",
"X-Auth-Key": "admin",
# "X-Auth-Token": "<X-Auth-Token>",
"objectUrl": "http://<ip_address>:8080/v1/AUTH_admin/container1/DLP.docx",
}
},
{
"user": "cloudtest\\clouduser",
"filename": "file1",
"queryID": "<queryID2>",
"uploadtype": "swift",
"objectInfo": {
"authUrl": "http://172.22.117.120:8080/auth/v1.0 ",
"X-Auth-User": "admin:admin",
"X-Auth-Key": "admin",
# "X-Auth-Token": "<X-Auth-Token>",
"objectUrl": "http://<ip_address>:8080/v1/AUTH_admin/container1/skyguard.txt",
},
},
{
"user": "cloudtest\\clouduser",
"filename": "file1",
"queryID": "<queryID3>",
"uploadtype": "s3",
"objectInfo": {
"s3Region": "cn-north-1",
"accessKey": "<AWS_AK>",
"secretKey": "<AWS_SK>",
"endpointUrl": "https://s3.cn-north-1.amazonaws.com.cn ",
"s3Bucket": "falseuser-test1",
"s3Key": "Desert.jpg",
},
}
]
}
r = requests.post(url, headers=auth_headers, json=data)
print(r.json())
Java Request Example - Batch Inspection
The following example shows how to use Java code to call the interface to batch upload multiple files in the WebService channel for content security review.
import java.util.HashMap;
import java.util.Iterator;
import java.io.File;
import java.io.FileInputStream;
import javax.net.ssl.SSLContext;
import java.security.cert.X509Certificate;
import com.alibaba.fastjson.JSONObject;
import com.alibaba.fastjson.JSONArray;
import org.apache.http.HttpEntity;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.ssl.SSLContextBuilder;
import org.apache.http.conn.ssl.TrustStrategy;
import org.apache.http.conn.ssl.TrustSelfSignedStrategy;
import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
import org.apache.http.conn.ssl.NoopHostnameVerifier;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.entity.ContentType;
import org.apache.http.entity.StringEntity;
import org.apache.http.util.EntityUtils;
public class TestCloudBulkAsync {
public static String inspectCloudBulk() {
String host = "<UCWI_HOST>";
String port = "<UCWI_PORT>";
String apiPath = "/skg/v1/dlp/channel/webserviceapp/78217a43-542d-47e2-97a5-4435bf11521d/bulk_async";
String url = "https://" + host + ":" + port + apiPath;
JSONObject bodyJson = new JSONObject();
JSONArray metadataArray = new JSONArray();
bodyJson.put("callback_url", "<CALLBACK_URL>");
bodyJson.put("verbose", true);
JSONObject metadataJson1 = new JSONObject();
JSONObject objectInfo1 = new JSONObject();
String queryId1 = UUID.randomUUID().toString();
metadataJson1.put("queryID", queryId1);
metadataJson1.put("user", "cloudtest\\clouduser");
metadataJson1.put("filename", "file1");
metadataJson1.put("uploadtype", "swift");
objectInfo1.put("authUrl", "http://172.22.117.120:8080/auth/v1.0 ");
objectInfo1.put("X-Auth-User", "admin:admin");
objectInfo1.put("X-Auth-Key", "admin");
objectInfo1.put("objectUrl", "http://172.22.117.120:8080/v1/AUTH_admin/container1/DLP.docx ");
metadataJson1.put("objectInfo", objectInfo1);
metadataArray.add(metadataJson1);
JSONObject metadataJson2 = new JSONObject();
JSONObject objectInfo2 = new JSONObject();
String queryId2 = UUID.randomUUID().toString();
metadataJson2.put("queryID", queryId2);
metadataJson2.put("user", "cloudtest\\clouduser");
metadataJson2.put("filename", "file2");
metadataJson2.put("uploadtype", "swift");
objectInfo2.put("authUrl", "http://<ip_address>:8080/auth/v1.0");
objectInfo2.put("X-Auth-User", "admin:admin");
objectInfo2.put("X-Auth-Key", "admin");
objectInfo2.put("objectUrl", "http://<ip_address>:8080/v1/AUTH_admin/container1/skyguard.txt");
metadataJson2.put("objectInfo", objectInfo2);
metadataArray.add(metadataJson2);
JSONObject metadataJson3 = new JSONObject();
JSONObject objectInfo3 = new JSONObject();
String queryId3 = UUID.randomUUID().toString();
metadataJson3.put("queryID", queryId3);
metadataJson3.put("user", "cloudtest\\clouduser");
metadataJson3.put("filename", "file3");
metadataJson3.put("uploadtype", "s3");
objectInfo3.put("s3Region", "cn-north-1");
objectInfo3.put("accessKey", "<AWS_AK>");
objectInfo3.put("secretKey", "<AWS_SK>");
objectInfo3.put("endpointUrl", "<S3_ENDPOINT>");
objectInfo3.put("s3Bucket", "<S3_BUCKET>");
objectInfo3.put("s3Key", "<S3_KEY>");
metadataJson3.put("objectInfo", objectInfo3);
metadataArray.add(metadataJson3);
bodyJson.put("objects", metadataArray);
HttpPost post = new HttpPost(url);
post.addHeader("Authorization", "<Authorization>");
post.addHeader("x-skg-timestamp", "<timestamp>");
post.setHeader("Content-type", "application/json");
CloseableHttpClient client = HttpClients.createDefault();
try {
SSLContext sslContext = new SSLContextBuilder().loadTrustMaterial(null, new TrustStrategy() {
public boolean isTrusted(X509Certificate[] chain, String authType) {
return true;
}
}).build();
SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(sslContext, new NoopHostnameVerifier());
client = HttpClients.custom().setSSLSocketFactory(sslsf).build();
}
catch (Exception e){
e.printStackTrace();
}
String bodyString = bodyJson.toJSONString();
StringEntity requestEntity = new StringEntity(bodyString,"utf-8");
post.setEntity(requestEntity);
String result = "";
try {
CloseableHttpResponse resp = client.execute(post);
HttpEntity entity = resp.getEntity();
result = EntityUtils.toString(entity, "UTF-8");
}
catch (Exception e){
e.printStackTrace();
}
return result;
}
public static void main(String []args) {
String result = inspectCloudBulk();
System.out.println(result);
}
}
Return Parameters
The bulk review request return result includes the following parameters:
| Name | Description |
|---|---|
| result | Whether the request is successful, 0 for success, 1 for failure |
| actionCode | When the request is successful, users can choose to perform default operations on content that meets the request conditions, including 1 - allow data transmission and 2 - block data transmission |
| errorCode | The error code returned when the request fails |
| message | The error message returned when the request fails |
| Incident_info | The event information of policy matching. If a policy is matched and non-compliant content is found, the system will also return policy matching information. Refer to the return example for detailed information. |
Return Example
If a policy is matched and non-compliant content is found, the system will also return policy matching information.
The following example and its notes provide a detailed introduction to each parameter in the return result and its meaning.
{
"cd2fd109-c4d4-489f-9b27-53752f7827d": {
"localDetectedTime": "2019-07-29T16:17:43.079368+0800", // Detection time
"incident_info": // Event information
{
"matchedPolicies": [ // Matched policies
{
"numberOfMatches": 1, // Number of events matched by the policy
"name": "abc\enduser1", // Domain and username of the违规用户 (non-compliant user)
"actionSettingName": "Block", // Action to be taken for non-compliant events
"matchedRules": [ // Matched policy rules
{
"name": "company", // Rule name
"matchedConditions": [ // Matched conditions
{
"type": 5, // Condition match type, including: 1: regex, 2: dictionary, 3: external script, 4: file type group, 5: keyword, 6: script, 7: file fingerprint, 8: machine learning, 9: terminal location, 10: file name, 11: attachment count, 12: database fingerprint, 13: file size, 14: binary, 15: compression file depth, 16: encrypted file, 17: format mismatch file, 18: built-in template
"matchedElements": [ // Matched elements
{
"matchedContents": [ // Matched content
{
"detectedValues": [ // Detected content
{
"text": "company" // Keyword
}
],
"isFileSuffixMatch": true, // Whether the file suffix matches
"isArchiveFile": false, // Whether it is a compressed file
"isEncryptFile": false, // Whether it is an encrypted file
"encodeType": "UnknownEncoding", // File encoding type, for details please refer to international standard encoding types and their explanations
"numberOfMatches": 1, // Number of matches
"locationPath": "confidential.doc", // Full path of the file
"id": "0-0", // File ID
"contentSize": 22 // Content size
}
],
"numberOfMatches": 1,
"isTruncated": false
}
],
"isTraditionalMatching": false // Whether traditional matching is enabled
}
]
}
],
"priority": 31, // Priority
"groupName": "Default Policy Group", // Policy group name
"severity": 3 // Sensitivity level 1 - High, 2 - Medium, 3 - Low, 4 - Information
}
]
},
"result": 0,
"actionCode": 2
},
"47992d42-04b7-4860-b186-b8c11f8b2253": {
"localDetectedTime": "2019-07-29T07:52:26.423684",
"incident_info": {},
"result": 0,
"actionCode": 1
}
}