Request for testers, linters, and critique on DF data files

[Dammann]

I think the remaining problems (with the Mentalist Power-Ups template) are now fixed.

Also: most of the monsters are now there, as well as an Alternate Abilities option for the Mentalist template.

[tmedwards]

I just ran the updated files and here's the result (files with issues: GURPS Dungeon Fantasy - Pyramid 3-36.gdf, GURPS Dungeon Fantasy - Pyramid 3-50.gdf, GURPS Dungeon Fantasy 14 - Psi.gdf).

Download: df_gdflint_report_20130819.txt

[Dammann]

The Pyramid 3-36 and 3-50 files are cleaned up, gotta finish up the DF 14 Psi one yet.

[tmedwards]

Not quite. GURPS Dungeon Fantasy - Pyramid 3-36.gdf is still not properly UTF-8 encoded.

Third try's the charm? Or, I could do that for you, if you're having trouble with it.

[Dammann]

I thought I'd done it twice- and now once more. If I still haven't done it properly, would you do so?

[tmedwards]

Sure.

Okay, looking at the file in depth now, this thing is a mishmash of various encodings. This is going to take some time to fix.

Where did you get some of the description() and modifiers() text from? A PDF to text extractor or something?

[tmedwards]

Alright, GURPS Dungeon Fantasy - Pyramid 3-36.gdf is UTF-8 encoded now and I'm pretty sure I got all of the mojibake out. I just updated the download on the repo, so you'll want to download it from there to update your local copy.

Oh, I also reformatted the author section a bit, mostly to include non-ASCII characters (specifically the registration and copyright marks). That is something I recommend as most text editors do not check the entire file when attempting to autodetect encodings, but only a small window at the top of the file. So, if you don't have an encoded character within the editor's detection window, it will misidentify the encoding and shenanigans will ensue.

[Dammann]

I think that some of the modifiers came from other data files, but I am sure that the descriptions include some text from pdf files. I am surprised this hasn't been an issue with the DF 14 file- I have done a bit of the same there.

[Dammann]

Should have started with this- thank you for your help, tmedwards!

[tmedwards]

As I noted, it was a mishmash of character encodings. It contained both Windows-1252 and UTF-8, which was probably the result of copy-pasting from multiple sources, which were in different encodings, without first normalizing them encoding-wise. Perhaps something to keep an eye on in the future. Anyway, the encoding SNAFU was likely why your efforts to UTF-8 encode the file were stymied.

The question about the description() and modifiers() text was because there were, still are in some (most?) cases, things like curly-/smart-quotes, em-/en-dashes, etc. which is generally a sign that the text was ripped from a PDF.