Script Locking up with doing more than one surface

[Krrrrrrk]

Script runs flawlessly when it's just Nauvis, but adding any more surfaces is when it freezes.

python auto.py --no-tags --surface=nauvis --surface="Nauvis Orbit"
checking for updates
No save name passed. Using most recent save: STREAM
factorio path: C:\Program Files (x86)\Steam\steamapps\common\Factorio\bin\x64\factorio.exe
output folder: script-output\FactorioMaps\STREAM
Enabling FactorioMaps mod
cleaning up
Building autorun.lua
Building config.ini
starting factorio
using steam launch hack
Processing STREAM/87/nauvis/day (1 of 4)
Surface capture STREAM/87/nauvis/day
crop 100.0% [=========================================================================================================]
ref 100.0% [=========================================================================================================]
generating thumbnail
zoom 100.0% [=========================================================================================================]
Processing STREAM/87/Nauvis Orbit/day (2 of 4)

Won't move past that. Let me know any other information you may need. Thank you.

[L0laapk3]

Hmm strange. Have you looked at the factorio log(s)? Are you positive that the surface name is correct (including capitalisation?)

[Krrrrrrk]

Logs not showing much, just normal stuff, but it's all from when I ran the game last - not showing anything when running this script. Names are proper, reran and ensured capitalization matches. Just for kicks ran it with the wrong capitalization, and the mod errored out, saying that surface didn't exist.

Attached is the full verbose when ran with all surfaces proper, was too long for a post.

krrrrrrk-verbose-log.txt

[L0laapk3]

can you attach your savefile & mod folder & mention which factorio version you're on? I'll see if I can find the time to try and reproduce it.

[Krrrrrrk]

1.1.80-0 build 60618 win64
Files are too big to attach, got them here:

Mods: https://mega.nz/file/UVURzZ6S#EkikZAQ8YakPnGlRywz-PeG7ulq46KVGVGusviowYME
Save: https://mega.nz/file/8d0jCBpT#ym7jru3CpsJwF-oaQQIMLbEJg8UC1wbHC1ABPHY2jDo

[Krrrrrrk]

Tried doing a fresh multi-surface save, locks up at the same place. I feel like it's trying to do something, I can hear my PC fans ramp up every few minutes when I just leave it be. Left it for almost an hour and no change, still stuck at the Processing stream-test/127/Nauvis Orbit/day (2 of 2).

Related to this - trying to do surfaces seems to have bricked any future snapshots as well for the save above. It's failing to find a "crop.txt" because... well it was never created for that snapshot. Any suggestions on how to remove that "hour 87" snapshot so it builds off the last non-bricked snapshot?

Traceback (most recent call last): File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\threading.py", line 980, in _bootstrap_inner self.run() File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\threading.py", line 917, in run self._target(*self._args, **self._kwargs) File "C:\Users\Krrrrrrk\AppData\Roaming\Factorio\mods\L0laapk3_FactorioMaps_4.3.0\auto.py", line 703, in refZoom ref(outFolder, timestamp, surface, daytime, args.basepath, args) File "C:\Users\Krrrrrrk\AppData\Roaming\Factorio\mods\L0laapk3_FactorioMaps_4.3.0\ref.py", line 194, in ref readCropList(os.path.join(topPath, "Images", data["maps"][old]["path"], surfaceName, daytime, "crop.txt"), False) File "C:\Users\Krrrrrrk\AppData\Roaming\Factorio\mods\L0laapk3_FactorioMaps_4.3.0\ref.py", line 175, in readCropList with open(path, "r", encoding="utf-8") as f: FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\Krrrrrrk\\AppData\\Roaming\\Factorio\\script-output\\FactorioMaps\\STREAM\\Images\\87\\nauvis\\day\\crop.txt'

[Krrrrrrk]

Just testing a few more things -
Made a new save, with my character on the surface of "Nauvis Orbit",

First test: I didn't specify any surfaces, based on the documentation was expecting it to do Nauvis Orbit since I was on that surface when I saved, but it just did nauvis.

Second Test: Used --surface="Nauvis Orbit" | Worked correctly, grabbed that surface and just that surface.

[Krrrrrrk]

I was able to get it working a bit.
I restored the STREAM FactorioMap folder from before I tried to do multi-surface, so I can at least take new snapshots of nauvis. Don't need to worry about fixing the bricked snapshot anymore.

I tried doing the surfaces one at a time to a new output folder, and that worked.

python auto.py gd1 STREAM --surface=nauvis --no-tags --verbose
python auto.py gd1 STREAM --surface="Nauvis Orbit" --no-tags --verbose
python auto.py gd1 STREAM --surface="Calidus Asteroid Belt 1" --no-tags --verbose

I can't do new surfaces to my existing timeline though. Get some variation of this:

Disabling FactorioMaps mod
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\multiprocessing\pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\multiprocessing\pool.py", line 48, in mapstar
    return list(map(*args))
  File "C:\Users\Krrrrrrk\AppData\Roaming\Factorio\mods\L0laapk3_FactorioMaps_4.3.0\ref.py", line 46, in compareRenderbox
    testResult = test((newPath, os.path.join(basePath, renderbox[1], renderbox[0]) + outext))
  File "C:\Users\Krrrrrrk\AppData\Roaming\Factorio\mods\L0laapk3_FactorioMaps_4.3.0\ref.py", line 18, in test
    newImg = Image.open(paths[0], mode='r').convert("RGB")
  File "C:\Users\Krrrrrrk\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\PIL\Image.py", line 3236, in open
    fp = builtins.open(filename, "rb")
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\Krrrrrrk\\AppData\\Roaming\\Factorio\\script-output\\FactorioMaps\\STREAM\\Images\\139\\Factory floor 1\\day\\renderboxes\\20\\-24_-24_24_24.png'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\Krrrrrrk\AppData\Roaming\Factorio\mods\L0laapk3_FactorioMaps_4.3.0\auto.py", line 904, in <module>
    auto(*sys.argv[1:])
  File "C:\Users\Krrrrrrk\AppData\Roaming\Factorio\mods\L0laapk3_FactorioMaps_4.3.0\auto.py", line 723, in auto
    refZoom()
  File "C:\Users\Krrrrrrk\AppData\Roaming\Factorio\mods\L0laapk3_FactorioMaps_4.3.0\auto.py", line 703, in refZoom
    ref(outFolder, timestamp, surface, daytime, args.basepath, args)
  File "C:\Users\Krrrrrrk\AppData\Roaming\Factorio\mods\L0laapk3_FactorioMaps_4.3.0\ref.py", line 351, in ref
    resultList = pool.map(partial(compareRenderbox, basePath=os.path.join(topPath, "Images"), new=str(newMap["path"])), compareList, 16)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\multiprocessing\pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\multiprocessing\pool.py", line 771, in get
    raise self._value
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\Krrrrrrk\\AppData\\Roaming\\Factorio\\script-output\\FactorioMaps\\STREAM\\Images\\139\\Factory floor 1\\day\\renderboxes\\20\\-24_-24_24_24.png'

When I check that folder I see a .jpg file there but not a .png.

[Krrrrrrk]

I've built a work around. Wrote a script to recursively check for .jpgs in "*\script-output\FactorioMaps\STREAM\images" while excluding any folders named "nauvis" (since this jpg/png issue only appears to be affecting factorissimo)

jpgtopng.py

import os
import shutil
from PIL import Image

def jpg_to_png(input_folder, exclude_folder):
    for root, dirs, files in os.walk(input_folder):
        # Remove the excluded folder 
        if exclude_folder in dirs:
            dirs.remove(exclude_folder)
        
        for file in files:
            if file.lower().endswith('.jpg'):
                jpg_file_path = os.path.join(root, file)
                png_file_path = os.path.splitext(jpg_file_path)[0] + '.png'

                # Convert time
                try:
                    with Image.open(jpg_file_path) as img:
                        img.save(png_file_path)
                    print(f"Converted: {jpg_file_path} -> {png_file_path}")
                except Exception as e:
                    print(f"Error converting {jpg_file_path}: {e}")

if __name__ == "__main__":
    folder_to_convert = r"C:\Users\Krrrrrrk\AppData\Roaming\Factorio\script-output\FactorioMaps\STREAM\images"
    folder_to_exclude = "nauvis"
    jpg_to_png(folder_to_convert, folder_to_exclude)

Not sure if it'll help you diagnose the underlying issue, but this is my current workflow.

• Save game on a surface that is not nauvis. (Not sure if this is required but seems to have helped.)
• Run the following commands, waiting for the first command to finish before starting the next:

• python auto.py --surface=nauvis --no-tags --verbose
• python jpgtopng.py
• python auto.py --surface="Nauvis Orbit" --no-tags --verbose
• python jpgtopng.py
• python auto.py --surface="Calidus Asteroid Belt 1" --no-tags --verbose
• python jpgtopng.py

After each auto.py it'll still throw errors about not finding the proper .png files, however as long as I run that conversion script when I do the next auto.py the index.html gets built properly. I'm not seeing any issues with the maps itself either, no improper chunks or images.

[L0laapk3]

Hey, I've been looking at this.

Been trying to recreate your minimal example. Can you walk me trough your mod selection a little bit?

Made a new save, with my character on the surface of "Nauvis Orbit"

How do you go about doing this? I created a new savefile with your mods, but I initially only the default nauvis surface seems to exist.

[Krrrrrrk]

Hey, I've been looking at this.

Been trying to recreate your minimal example. Can you walk me trough your mod selection a little bit?

Made a new save, with my character on the surface of "Nauvis Orbit"

How do you go about doing this? I created a new savefile with your mods, but I initially only the default nauvis surface seems to exist.

I might have worded that badly. If I recall correctly (it's been a bit), it was the same "worlds" just a new save file, that way the script didn't need to do anything with snapshots or history.

[L0laapk3]

Yeah thats what I was trying to do as well to set up a minimal example, since the savefile you provided is not exactly small :P

However when I created a new world with your mods, only one surface existed and I'm unfamiliar with the mods you used