Enabling FileSystemWatcher without AutoUpdate

[ollyburns]

Hi,

I run quite a large estate of containers. Following your documentation, I have an offline process to download the hash file, so we don't overload the download service. I have the following pipeline configuration in place which, I think / hope, watches the file system for changes and updates the copied and in memory file when the shared file is updated.

I have two questions:

1. Is the CreateTempDataCopy necessary given the file is all loaded into memory when MaxPerformance is selected? The reason I ask this is that I also use the same technique for IP Location data and the file is massive, so I see big iops spikes while the file is being copied.
2. This config generates regular errors in the logs like this: HTTP status code '400' from data update service at 'https://distributor.51degrees.com/api/v2/download?LicenseKeys=&Download=True&Type=HashV41', and I was wondering whether there is a way of supressing this auto fetch via config?

<?xml version="1.0" encoding="utf-8" ?>
<PipelineOptions>
    <Elements>
        <Element>
            <BuildParameters>
                <DataFile>/mnt/data/Enterprise-HashV41.hash</DataFile>
                <PerformanceProfile>MaxPerformance</PerformanceProfile>
                <CreateTempDataCopy>true</CreateTempDataCopy>
                <AutoUpdate>true</AutoUpdate>
                <DataFileSystemWatcher>true</DataFileSystemWatcher>
                <UpdatePollingIntervalSeconds>1800</UpdatePollingIntervalSeconds>
            </BuildParameters>
            <BuilderName>DeviceDetectionHashEngine</BuilderName>
        </Element>
   </Elements>
</PipelineOptions>

We are using the DataUpdateServiceDefault, set up like this:

DataUpdateService dataUpdateService = new DataUpdateServiceDefault();
File optionsFile = FileFinder.getFilePath("DeviceEnricherPipeline.xml");
PipelineOptions pipelineOptions = PipelineOptionsFactory.getOptionsFromFile(optionsFile);
Pipeline pipeline = new FiftyOnePipelineBuilder().addService(dataUpdateService).buildFromConfiguration(pipelineOptions);

Thank you for your help.

[shaitanaleksander]

Hi,

Thank you for your detailed question. I'll address both points:

Question 1: Is CreateTempDataCopy necessary with MaxPerformance?

No, it's not strictly necessary in your setup. Since you're using MaxPerformance profile, the entire file is read
into memory. You can set CreateTempDataCopy=false to eliminate the IOPS spikes across your container fleet.

Just ensure your offline download process uses atomic file replacement:

wget -O /tmp/Enterprise-HashV41.hash.tmp <url>
mv /tmp/Enterprise-HashV41.hash.tmp /mnt/data/Enterprise-HashV41.hash

The DataFileSystemWatcher will detect the file change and trigger a reload into memory.

Question 2: Suppressing the auto-update HTTP 400 errors

The pipeline requires AutoUpdate=true to enable the DataFileSystemWatcher functionality. Unfortunately, it will also
attempt polling even when you only intend to use file system watcher updates.

While it's not ideal that we cannot enable only one mode or the other, as a workaround you can suppress the polling
by:

• Setting a very large UpdatePollingIntervalSeconds value, or
• Not providing a DataUpdateLicenseKey

This way the service will not attempt to make those HTTP calls.

Please let us know if you have any further questions.

[ollyburns]

Thank you for your response.
Not providing DataUpdateLicenseKey does seem to supress the auto update URL being called for device-detection, but not ip-intelligence.