LogSID: Detecting Sensitive Information
in Large-Scale Logs

Anonymous Authors

LogSID detects sensitive text in individual log messages by combining a learned cache, a small language model, and selective LLM detection. The study constructs a benchmark of 32,983,908 messages from eight sources and evaluates character-level sensitive-span detection on chronologically held-out logs.

Source code is available on GitHub. Datasets are available on Zenodo.

Abstract

Logs support troubleshooting and analysis across teams, but sharing them can expose sensitive information. Similar structures can contain different sensitive spans or none at all, making field-format matching or one label assignment per template insufficient. LogSID reuses content-dependent span-localization rules through a learned cache. A small language model handles messages without an accepted cache result; an LLM is invoked only when the small model's complete prediction is not accepted. Historical annotations initialize the cache, train the small model, and calibrate acceptance rules, while online cache updates support subsequent messages without retraining the small model.

What is the task in LogSID?

Identify the sensitive text and its categories in each log message. The example below shows why two messages with the same pattern can require different predictions.

Two BGL messages share a pattern. Only the pseudonym userA in the personal path is annotated as an account identifier. The shared application path has no sensitive annotation.
Figure 1. Two BGL messages with the same pattern but different gold labels. Only userA is sensitive in (a); (b) has no annotated sensitive text. The account name is pseudonymized. Full-size figure ↗

How does LogSID work?

A message first reaches the detection cache. Only messages without an accepted cache or small-model result proceed to the next stage. Checked model outputs can update the cache for subsequent messages.

The LogSID method: historical initialization supports a content-dependent detection cache, small language model detection, and selective LLM detection. Accepted predictions produce sensitive spans or an empty set; checked outputs update the cache.
Figure 2. Overview of LogSID. After matching structure, the cache selects span rules from the current content. Solid arrows show detection; dashed arrows show preparation and updates. Full-size figure ↗

The LogSID Benchmark

The benchmark covers eight log sources and annotates sensitive text in both fixed wording and variable values. The taxonomy defines 18 categories; 12 occur in the collected data.

Table 1. Overview of the eight-source benchmark.

Scroll horizontally to view all columns →

SourceLogsTemplatesSensitive (%)CategoriesMixedVarying
Zanbil10,365,15211100.00906
Apache51,9782959.81200
AIT-mail382,1172797.81411
OpenSSH638,9473894.06309
HealthApp212,39415683.99400
Mac100,3146264.94538
BGL4,631,2613203.0421111
Thunderbird16,601,7451,24120.615418
Total32,983,9082,44845.83121953

Sensitive (%): messages with at least one sensitive annotation. Categories: observed categories; the total is the union across sources. Mixed: templates containing both sensitive and non-sensitive messages. Varying: templates with varying category presence. AIT-mail contains simulated-user activity on real services in a testbed.

Experimental Results

Evaluation uses the chronologically held-out 80% of each source: 26,387,130 test messages in total. Precision, recall, and F1 require strict agreement on span boundaries and category.

Table 2. Detection effectiveness on the held-out test messages (%).

Scroll horizontally to view all columns →

SourceSDLogUNLEASHGLiNER-PIIPresidioLogSID
PRF1PRF1PRF1PRF1PRF1
Zanbil99.25195.74097.46494.23792.16493.189———7.18834.73411.91199.97498.34999.155
Apache99.99499.99899.99699.64799.92399.7850.0000.0000.00075.56450.00060.18099.97799.98199.979
AIT-mail99.52476.11486.25999.63076.11486.2998.4533.6985.1452.3881.2841.67099.58396.74398.142
OpenSSH98.86854.34270.13588.08147.97562.1176.6292.9064.04161.52244.83651.87098.93899.94799.440
HealthApp98.71999.04798.88385.44375.42480.1210.0000.0000.0000.0000.0000.00099.93796.50198.189
Mac52.10859.12055.39352.44646.59049.3455.77714.3848.2433.03151.7185.72798.98888.92593.687
BGL0.0000.0000.00010.0629.0099.5070.0000.0000.0000.0000.0000.00061.57584.31871.174
Thunderbird98.91382.04989.6957.12616.4679.9471.2161.3631.2850.1190.2000.14999.98274.61885.458
Mean80.92270.80174.72867.08457.95861.2893.1543.1932.67418.72722.84716.43894.86992.42393.153

Bold and underlined values indicate the best and second-best results. Mean P/R/F1 are averaged separately with equal weight per source. GLiNER-PII includes seven available sources; other methods include all eight. — denotes an evaluation excluded because full-test detection was projected to exceed six hours; it provides no measured timeout or score. SDLog and UNLEASH use the task adaptations evaluated in the paper.

LogSID achieves 93.15% mean source F1. Its advantage varies by workload: SDLog is stronger on Apache, HealthApp, and Thunderbird, while LogSID improves over SDLog on the other five sources.

Context-sensitive decisions

Strict-span F1 comparison between SDLog and LogSID for mixed-sensitivity and changing-category groups from the same test messages.
Figure 3. SDLog and LogSID on the same test groups. Higher strict-span F1 is better. Groups may overlap; groups with no test messages are omitted. Full-size figure ↗

Detection time

Average measured detection time across eight test sources, in seconds: LogSID 412.8, SDLog 2854.4, UNLEASH 3140.1, and Presidio 9292.9.
Figure 4. Average detection time across eight sources. All four methods report measured mean durations. Full-size figure ↗