134. Salesforce
The Salesforce add-on provides support for fetching Event Log Files from Salesforce with NXLog. The script collects Event Log Files from a Salesforce instance by periodically running SOQL queries via the REST API. The Events can then be passed to NXLog by different means, depending how the data collection is configured.
For more information about the Event Log File API, see EventLogFile in the Salesforce SOAP API Developer Guide.
Note
|
The Event Logs feature of Salesforce is a a paid add-on feature. Make sure this feature is enabled on the Salesforce instance before continuing. |
134.1. General Usage
The salesforce.py
script can be configured both from the command line and
from
a configuration file. The configuration file collect.conf.json
must be located in the same directory as salesforce.py
, so that
the script can load the configuration parameters automatically.
Passing arguments from the command line overrides the corresponding
parameter read from the configuration file. The following is a
sample configuration file:
{
"log_level": "DEBUG",
"log_file": "var/collector.log",
"user": "user@example.com",
"password": "UxQqx847sQ",
"token": "ZsQO0k5gAgJch3mLUtEqt0K",
"url": "https://login.salesforce.com/services/Soap/u/39.0/",
"checkpoint": "var/checkpoint/",
"keep_csv": "True",
"output": "structured",
"header": "none",
"mode": "across",
"transport": "stdout",
"target": "file",
"limit": "5",
"delay": "3",
"request_delay": "3600"
}
A compact view of the command line options is shown below. Use
salesforce.py -h
to get help, including a short explanation
of the options.
usage: salesforce.py [-h] [--config CONFIG] [--user USER]
[--password PASSWORD] [--token TOKEN] [--url URL]
[--checkpoint CHECKPOINT] [--keep_csv {True,False}]
[--output {json,structured}] [--header {none,syslog}]
[--mode {loop,across}] [--target TARGET] [--delay DELAY]
[--limit LIMIT] [--request_delay REQUEST_DELAY]
[--transport {file,socket,pipe,stdout}]
[--log_level {CRITICAL,ERROR,WARNING,INFO,DEBUG,NOTSET}]
[--log_file LOG_FILE]
134.2. Authentication and Data Retrieval
The user needs to set the authentication parameters (username
, password
,
and token
) so that the script can connect to Salesforce and retrieve the
Event Logs. The url
parameter supplied with the sample configuration file
is correct at the time of writing but it
may change in the future. The log_level
and log_file
parameters can be used as an aid during the initial setup, as well as to
identify problems during operation.
Note
|
It is not possible to find the security token of an existing profile. The solution is to reset it as described in Reset Your Security Token on Salesforce Help. |
Depending on your setup, the mode
parameter can be set to loop
so that the
script will look for new events continuously or to across
so that once all
the available events are retrieved the script will terminate. When in loop
mode, the request_delay
can be configured for the script to wait the
specified number of seconds before requesting more events.
134.3. Local Storage and Processing
The script will temporarily store the Event Log Files in a directory structure
under the directory name given by the checkpoint
parameter. The events are
stored in CSV format. Files with the same name but with a .state
extension hold the current state, so that no events will be lost or duplicated
even if the script terminates unexpectedly. The the directory structure is
shown below.
Note
|
Directories and files are created automatically when an event of that type is logged by Salesforce. |
var/checkpoint/ApexExecution: 2018-02-08T00:00:00.000+0000.csv 2018-02-08T00:00:00.000+0000.state var/checkpoint/LightningError: 2018-02-08T00:00:00.000+0000.csv 2018-02-08T00:00:00.000+0000.state var/checkpoint/Login: 2018-02-08T00:00:00.000+0000.csv 2018-02-08T00:00:00.000+0000.state var/checkpoint/Logout: 2018-02-08T00:00:00.000+0000.csv 2018-02-08T00:00:00.000+0000.state var/checkpoint/PackageInstall: 2018-03-01T00:00:00.000+0000.csv 2018-03-01T00:00:00.000+0000.state
Warning
|
If this directory structure is removed, the script will be unable to determine the state and all available events stored in your Salesforce instance will be retrieved and passed to NXLog again. However, after testing and determining that everything is configured correctly, remember to delete the directory structure to reset the state. |
Once all the available events have been downloaded and the script determines
that no other events has been added, it will proceed to process them and
produce the final output. The limit
and delay
parameters can be set
to throttle the processing by limiting by number of records and delaying between
blocks of records in seconds.
The script will delete the CSV files once those are processed. However, the
keep_csv
parameter can be set to True to preserve them.
134.4. Data Format and Transport
The processed events can be presented in two different formats: either
as structured output or as JSON. This can be selected by setting the
output
parameter accordingly. Furthermore, a Syslog style header can be
added before the event data by means of the header
parameter. The
output types are show below.
CLIENT_IP="46.198.211.113" OS_NAME="LINUX" DEVICE_SESSION_ID="33ddcf5f751fdaf4b6a010d73014710ed2f13e33" BROWSER_NAME="CHROME" BROWSER_VERSION="64" USER_AGENT=""Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36"" CLIENT_ID="" REQUEST_ID="" SESSION_KEY="qomr/wgmbMU73iG6" DEVICE_ID="" CONNECTION_TYPE="" EVENT_TYPE="LightningError" SDK_APP_VERSION="" SDK_APP_TYPE="" UI_EVENT_SOURCE="storage" SDK_VERSION="" UI_EVENT_SEQUENCE_NUM="" LOGIN_KEY="5ujU+09kPSKatTxR" UI_EVENT_TYPE="error" PAGE_START_TIME="1519928816975" DEVICE_MODEL="" USER_TYPE="Standard" ORGANIZATION_ID="00D1r000000rH0F" OS_VERSION="" USER_ID_DERIVED="0051r000007NyeqAAC" UI_EVENT_ID="ltng:error" APP_NAME="one:one" UI_EVENT_TIMESTAMP="1519928819334" USER_ID="0051r000007Nyeq" TIMESTAMP="20180301182702.187" TIMESTAMP_DERIVED="2018-03-01T18:27:02.187Z" DEVICE_PLATFORM="SFX:BROWSER:DESKTOP"
{"CLIENT_IP": "Salesforce.com IP", "REQUEST_ID": "4GVCi4pxSjCESP-qby__7-", "SESSION_KEY": "", "API_TYPE": "", "EVENT_TYPE": "Login", "SOURCE_IP": "46.198.211.113", "RUN_TIME": "143", "LOGIN_KEY": "", "USER_NAME": "user@example.com", "CPU_TIME": "57", "BROWSER_TYPE": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36", "URI": "/index.jsp", "ORGANIZATION_ID": "00D1r000000rH0F", "USER_ID_DERIVED": "0051r000007NyeqAAC", "DB_TOTAL_TIME": "47093446", "LOGIN_STATUS": "LOGIN_NO_ERROR", "USER_ID": "0051r000007Nyeq", "TIMESTAMP": "20180302083919.878", "TLS_PROTOCOL": "TLSv1.2", "REQUEST_STATUS": "", "CIPHER_SUITE": "ECDHE-RSA-AES256-GCM-SHA384", "TIMESTAMP_DERIVED": "2018-03-02T08:39:19.878Z", "URI_ID_DERIVED": "", "API_VERSION": "9998.0"}
<14>1 2018-03-05T18:37:56.157860 eu12.salesforce.com - - - - NUMBER_FIELDS="2" CLIENT_IP="46.198.211.113" ENTITY_NAME="EventLogFile" DB_CPU_TIME="0" USER_AGENT="5238" REQUEST_ID="4GUW0E969JxN49-qbzCo8-" SESSION_KEY="mmOUNLlL4HlSzrSq" EVENT_TYPE="RestApi" RUN_TIME="8" RESPONSE_SIZE="706" METHOD="GET" CPU_TIME="4" LOGIN_KEY="szBoBvcp+3dHeuff" STATUS_CODE="200" URI="/services/data/v37.0/sobjects/EventLogFile/0AT1r000000NWSKGA4/LogFile" ORGANIZATION_ID="00D1r000000rH0F" REQUEST_STATUS="S" DB_TOTAL_TIME="3319055" ROWS_PROCESSED="1" MEDIA_TYPE="text/csv" DB_BLOCKS="15" USER_ID="0051r000007Nyeq" TIMESTAMP="20180301190010.634" URI_ID_DERIVED="0AT1r000000NWSKGA4" REQUEST_SIZE="0" USER_ID_DERIVED="0051r000007NyeqAAC" TIMESTAMP_DERIVED="2018-03-01T19:00:10.634Z"
Note
|
The samples above are not from the same event. |
The formatted output can then be displayed in standard output, passed to
another program by a named pipe, saved to a file, or sent to another
program using Unix Domain Sockets (UDS). This can be controlled by setting the
transport
parameter to stdout, pipe, file, or socket respectively.
When the transport
is pipe, file, or socket the target
parameter
can be used to set the name of the pipe, file, or socket.
134.5. Configuring NXLog
The versatility of the salesforce.py
script, combined with NXLog, allows
for several different ways to collect the Event Log Files from Salesforce.
A first scenario is that NXLog is running the script directly and consumes the data from the script. To do this, the script should be running in loop mode, so that events are fetched periodically from Salesforce.
NXLog executes salesforce.py
, which in turn collects events every hour,
processes them, formats them as JSON with a Syslog header, and forwards them
to NXLog.
{
"log_level": "DEBUG",
"log_file": "var/collector.log",
"user": "user@example.com",
"password": "UxQqx847sQ",
"token": "ZsQO0k5gAgJch3mLUtEqt0K",
"url": "https://login.salesforce.com/services/Soap/u/39.0/",
"checkpoint": "var/checkpoint/",
"keep_csv": "True",
"output": "json",
"header": "syslog",
"mode": "loop",
"transport": "stdout",
"target": "file",
"limit": "100",
"delay": "3",
"request_delay": "3600"
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
<Extension _syslog>
Module xm_syslog
</Extension>
<Extension _json>
Module xm_json
</Extension>
<Input messages>
Module im_exec
Command ./salesforce.py
<Exec>
parse_syslog();
parse_json($Message);
</Exec>
</Input>
<Output out>
Module om_file
File "output.log"
</Output>
<Route messages_to_file>
Path messages => out
</Route>
A second scenario: set up NXLog to listen on a UDS for events and use either
NXLog or an external scheduler to run salesforce.py
. In this case,
salesforce.py
runs in across mode.
Warning
|
Be sure to provide ample time for the script to finish executing before the scheduler starts a new execution. Or use a shell script that prevents running multiple instances simultaneously. |
{
"log_level": "DEBUG",
"log_file": "var/collector.log",
"user": "user@example.com",
"password": "UxQqx847sQ",
"token": "ZsQO0k5gAgJch3mLUtEqt0K",
"url": "https://login.salesforce.com/services/Soap/u/39.0/",
"checkpoint": "var/checkpoint/",
"keep_csv": "True",
"output": "structured",
"header": "none",
"mode": "across",
"transport": "socket",
"target": "uds_socket",
"limit": "100",
"delay": "3",
"request_delay": "3600"
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
<Extension exec>
Module xm_exec
<Schedule>
Every 1 hour
<Exec>
log_info("Scheduled execution at " + now());
exec_async("./salesforce.py");
</Exec>
</Schedule>
</Extension>
<Input messages>
Module im_uds
UDS ./uds_socket
UDSType stream
</Input>
<Output out>
Module om_file
File "output.log"
</Output>
<Route messages_to_file>
Path messages => out
</Route>
It is even possible to manually start the salesforce.py
in loop mode with
a large request_delay
and collect via UDS (as shown above) without the
xm_exec instance. Or set the transport
to file and configure
NXlog to read events with im_file.
Note
|
Though events are captured in real time, Salesforce generates the Event Log Files during non-peak hours. |