Forum:Download the NetHackWiki - Revision history

Rengaf: rengaf is back

2025-07-17T14:26:33Z

rengaf is back

Dtype: /* Offline wiki */ showing link to download

2025-05-18T16:25:43Z

‎Offline wiki: showing link to download

Tim: /* Offline wiki */

2025-05-03T00:16:35Z

‎Offline wiki

Tmp6154: /* Offline wiki */

2024-08-22T21:07:55Z

‎Offline wiki

Winny: update on zim progress

2024-08-22T19:09:38Z

update on zim progress

Winny: /* Offline wiki */ zim!

2024-08-19T17:42:07Z

‎Offline wiki: zim!

KeyboardFire: didn't do that signature thing right

2015-06-19T07:37:47Z

didn't do that signature thing right

KeyboardFire at 07:36, 19 June 2015

2015-06-19T07:36:10Z

Quantum Immortal: /* Offline wiki */

2015-06-14T06:18:54Z

‎Offline wiki

Deadagain: /* Offline wiki */

2015-06-12T13:49:50Z

‎Offline wiki

← Older revision		Revision as of 14:26, 17 July 2025
Line 29:		Line 29:

	:::::If you're looking for reliability or speed, you don't need to scrape the live wiki. It has some bot restrictions these days due to the obscene number of AI training bots that absolutely hammer the site 24x7. You can download a complete copy here if you want to run a local mirror to scrape at much higher speed. [[NetHackWiki:Download]] [[User:Dtype\|dtype]] ([[User talk:Dtype\|talk]]) 16:25, 18 May 2025 (UTC)		:::::If you're looking for reliability or speed, you don't need to scrape the live wiki. It has some bot restrictions these days due to the obscene number of AI training bots that absolutely hammer the site 24x7. You can download a complete copy here if you want to run a local mirror to scrape at much higher speed. [[NetHackWiki:Download]] [[User:Dtype\|dtype]] ([[User talk:Dtype\|talk]]) 16:25, 18 May 2025 (UTC)
		+
		+	::::::Actually, scraping the live wiki is the most optimal solution, because of custom plugins such as SimpleTabs (which I could not make work even after hours of docker pain). I added my zim to [[NetHackWiki:Download]], it supports SimpleTabs and in my opinion looks better than what mwoffliner gives you. And for anyone who wants to see some code, [https://gitlab.com/wildangelcult/nethackwiki-scripts here are some scripts] I am using to make it. --[[User:Rengaf\|Rengaf]] ([[User talk:Rengaf\|talk]]) 14:26, 17 July 2025 (UTC)


	Oooh, thank you very much! Now I can throw away that ugly wiki scraper I had lying around. :D ——[[User:KeyboardFire\|KeyboardFire]] ([[User talk:KeyboardFire\|talk]]) 07:37, 19 June 2015 (UTC)		Oooh, thank you very much! Now I can throw away that ugly wiki scraper I had lying around. :D ——[[User:KeyboardFire\|KeyboardFire]] ([[User talk:KeyboardFire\|talk]]) 07:37, 19 June 2015 (UTC)

← Older revision		Revision as of 16:25, 18 May 2025
Line 27:		Line 27:

	::::I've been creating a Zim of the wiki for a while for my own purposes and I've tested it against the git version of Kiwix for android (as I'm a gurrhack player), but it regularly fails due to (I suspect getting banned or the server timing out, or just mwoffliner failing). The latest successful scrape and package is available for download here: https://nethack.0x1a.ro/nethackwiki_en_all_2025-01.zim --[[User:Tim\|Parted]] ([[User talk:Tim\|talk]]) 00:15, 3 May 2025 (UTC)		::::I've been creating a Zim of the wiki for a while for my own purposes and I've tested it against the git version of Kiwix for android (as I'm a gurrhack player), but it regularly fails due to (I suspect getting banned or the server timing out, or just mwoffliner failing). The latest successful scrape and package is available for download here: https://nethack.0x1a.ro/nethackwiki_en_all_2025-01.zim --[[User:Tim\|Parted]] ([[User talk:Tim\|talk]]) 00:15, 3 May 2025 (UTC)
		+
		+	:::::If you're looking for reliability or speed, you don't need to scrape the live wiki. It has some bot restrictions these days due to the obscene number of AI training bots that absolutely hammer the site 24x7. You can download a complete copy here if you want to run a local mirror to scrape at much higher speed. [[NetHackWiki:Download]] [[User:Dtype\|dtype]] ([[User talk:Dtype\|talk]]) 16:25, 18 May 2025 (UTC)


	Oooh, thank you very much! Now I can throw away that ugly wiki scraper I had lying around. :D ——[[User:KeyboardFire\|KeyboardFire]] ([[User talk:KeyboardFire\|talk]]) 07:37, 19 June 2015 (UTC)		Oooh, thank you very much! Now I can throw away that ugly wiki scraper I had lying around. :D ——[[User:KeyboardFire\|KeyboardFire]] ([[User talk:KeyboardFire\|talk]]) 07:37, 19 June 2015 (UTC)

@@ Line 6: / Line 6: @@
 == Offline wiki ==
 Thanks!
@@ Line 24: / Line 25: @@
 :::As I wrote on roguelikes discord, the problem lies in the absence of [https://www.mediawiki.org/wiki/Extension:VisualEditor VisualEditor]. The mwoffliner 1.13.0 doesn't detect this, but the latest commit of it does. It is also briefly discussed at https://github.com/openzim/mwoffliner/issues/1963. --rengaf/[[User:Tmp6154|Tmp6154]] ([[User talk:Tmp6154|talk]]) 21:07, 22 August 2024 (UTC)
 Oooh, thank you very much! Now I can throw away that ugly wiki scraper I had lying around. :D ——[[User:KeyboardFire|KeyboardFire]] ([[User talk:KeyboardFire|talk]]) 07:37, 19 June 2015 (UTC)

← Older revision		Revision as of 21:07, 22 August 2024
Line 22:		Line 22:

	:::We were discussing this matter on the [https://discord.gg/dWMkgHn roguelikes Discord] in #nethack. Turns out somebody did make a [https://github.com/romanthekat/nethackwiki-zim/releases/tag/2022-07 ZIM archive using mwoffliner back in 2022]. Unfortunately mwoffliner no longer works because of the aforementioned rest api endpoint unavailable for rendering HTML. In the meantime I did try [https://zimit.kiwix.org/ zimit], but the archive it generated wasn't complete (the official zimit instance mentions a limit on requests...). I'm still resuming the wget-2-zim script occasionally, oft triggering some sort of automatic firewall after 1-10 minutes fetches, then suspending the job in order to pick it up later. Any suggestions appreciated. [[User:Winny\|Winny]] ([[User talk:Winny\|talk]]) 19:09, 22 August 2024 (UTC)		:::We were discussing this matter on the [https://discord.gg/dWMkgHn roguelikes Discord] in #nethack. Turns out somebody did make a [https://github.com/romanthekat/nethackwiki-zim/releases/tag/2022-07 ZIM archive using mwoffliner back in 2022]. Unfortunately mwoffliner no longer works because of the aforementioned rest api endpoint unavailable for rendering HTML. In the meantime I did try [https://zimit.kiwix.org/ zimit], but the archive it generated wasn't complete (the official zimit instance mentions a limit on requests...). I'm still resuming the wget-2-zim script occasionally, oft triggering some sort of automatic firewall after 1-10 minutes fetches, then suspending the job in order to pick it up later. Any suggestions appreciated. [[User:Winny\|Winny]] ([[User talk:Winny\|talk]]) 19:09, 22 August 2024 (UTC)
		+
		+	:::As I wrote on roguelikes discord, the problem lies in the absence of [https://www.mediawiki.org/wiki/Extension:VisualEditor VisualEditor]. The mwoffliner 1.13.0 doesn't detect this, but the latest commit of it does. It is also briefly discussed at https://github.com/openzim/mwoffliner/issues/1963. --rengaf/[[User:Tmp6154\|Tmp6154]] ([[User talk:Tmp6154\|talk]]) 21:07, 22 August 2024 (UTC)

	Oooh, thank you very much! Now I can throw away that ugly wiki scraper I had lying around. :D ——[[User:KeyboardFire\|KeyboardFire]] ([[User talk:KeyboardFire\|talk]]) 07:37, 19 June 2015 (UTC)		Oooh, thank you very much! Now I can throw away that ugly wiki scraper I had lying around. :D ——[[User:KeyboardFire\|KeyboardFire]] ([[User talk:KeyboardFire\|talk]]) 07:37, 19 June 2015 (UTC)

← Older revision		Revision as of 19:09, 22 August 2024
Line 20:		Line 20:

	::ZIM would be nice ([https://wiki.openzim.org/wiki/Build_your_ZIM_file howto]). [http://xowa.org/ Xowa] doesn't work on my Debian Testing and seems unmaintained (last official release [https://github.com/gnosygnu/xowa/releases/ fall 2020]). ZIM appears to be a gold standard for offline website retrieval. I did attempt to use [https://wiki.openzim.org/wiki/Build_your_ZIM_file#MWoffliner mwoffliner] however the rest api endpoint does not work, so the script cannot download page HTML. As an alternative I tried [https://wiki.openzim.org/wiki/Build_your_ZIM_file#wget-2-zim wget-2-zim] but kept getting firewalled for whatever reason. We could benefit from ZIM. Wiki dumps are insufficient to ensure nethackwiki is accurately archived for offline use... [[User:Winny\|Winny]] ([[User talk:Winny\|talk]]) 17:42, 19 August 2024 (UTC)		::ZIM would be nice ([https://wiki.openzim.org/wiki/Build_your_ZIM_file howto]). [http://xowa.org/ Xowa] doesn't work on my Debian Testing and seems unmaintained (last official release [https://github.com/gnosygnu/xowa/releases/ fall 2020]). ZIM appears to be a gold standard for offline website retrieval. I did attempt to use [https://wiki.openzim.org/wiki/Build_your_ZIM_file#MWoffliner mwoffliner] however the rest api endpoint does not work, so the script cannot download page HTML. As an alternative I tried [https://wiki.openzim.org/wiki/Build_your_ZIM_file#wget-2-zim wget-2-zim] but kept getting firewalled for whatever reason. We could benefit from ZIM. Wiki dumps are insufficient to ensure nethackwiki is accurately archived for offline use... [[User:Winny\|Winny]] ([[User talk:Winny\|talk]]) 17:42, 19 August 2024 (UTC)
		+
		+	:::We were discussing this matter on the [https://discord.gg/dWMkgHn roguelikes Discord] in #nethack. Turns out somebody did make a [https://github.com/romanthekat/nethackwiki-zim/releases/tag/2022-07 ZIM archive using mwoffliner back in 2022]. Unfortunately mwoffliner no longer works because of the aforementioned rest api endpoint unavailable for rendering HTML. In the meantime I did try [https://zimit.kiwix.org/ zimit], but the archive it generated wasn't complete (the official zimit instance mentions a limit on requests...). I'm still resuming the wget-2-zim script occasionally, oft triggering some sort of automatic firewall after 1-10 minutes fetches, then suspending the job in order to pick it up later. Any suggestions appreciated. [[User:Winny\|Winny]] ([[User talk:Winny\|talk]]) 19:09, 22 August 2024 (UTC)

	Oooh, thank you very much! Now I can throw away that ugly wiki scraper I had lying around. :D ——[[User:KeyboardFire\|KeyboardFire]] ([[User talk:KeyboardFire\|talk]]) 07:37, 19 June 2015 (UTC)		Oooh, thank you very much! Now I can throw away that ugly wiki scraper I had lying around. :D ——[[User:KeyboardFire\|KeyboardFire]] ([[User talk:KeyboardFire\|talk]]) 07:37, 19 June 2015 (UTC)

← Older revision		Revision as of 17:42, 19 August 2024
Line 18:		Line 18:

	:OpenZIM sounds like a good idea. What about providing a torrent too? With a RSS torrent feed, we could even update it automatically. --[[User:Quantum Immortal\|Quantum Immortal]] ([[User talk:Quantum Immortal\|talk]]) 06:18, 14 June 2015 (UTC)		:OpenZIM sounds like a good idea. What about providing a torrent too? With a RSS torrent feed, we could even update it automatically. --[[User:Quantum Immortal\|Quantum Immortal]] ([[User talk:Quantum Immortal\|talk]]) 06:18, 14 June 2015 (UTC)
		+
		+	::ZIM would be nice ([https://wiki.openzim.org/wiki/Build_your_ZIM_file howto]). [http://xowa.org/ Xowa] doesn't work on my Debian Testing and seems unmaintained (last official release [https://github.com/gnosygnu/xowa/releases/ fall 2020]). ZIM appears to be a gold standard for offline website retrieval. I did attempt to use [https://wiki.openzim.org/wiki/Build_your_ZIM_file#MWoffliner mwoffliner] however the rest api endpoint does not work, so the script cannot download page HTML. As an alternative I tried [https://wiki.openzim.org/wiki/Build_your_ZIM_file#wget-2-zim wget-2-zim] but kept getting firewalled for whatever reason. We could benefit from ZIM. Wiki dumps are insufficient to ensure nethackwiki is accurately archived for offline use... [[User:Winny\|Winny]] ([[User talk:Winny\|talk]]) 17:42, 19 August 2024 (UTC)

	Oooh, thank you very much! Now I can throw away that ugly wiki scraper I had lying around. :D ——[[User:KeyboardFire\|KeyboardFire]] ([[User talk:KeyboardFire\|talk]]) 07:37, 19 June 2015 (UTC)		Oooh, thank you very much! Now I can throw away that ugly wiki scraper I had lying around. :D ——[[User:KeyboardFire\|KeyboardFire]] ([[User talk:KeyboardFire\|talk]]) 07:37, 19 June 2015 (UTC)

← Older revision		Revision as of 07:37, 19 June 2015
Line 19:		Line 19:
	:OpenZIM sounds like a good idea. What about providing a torrent too? With a RSS torrent feed, we could even update it automatically. --[[User:Quantum Immortal\|Quantum Immortal]] ([[User talk:Quantum Immortal\|talk]]) 06:18, 14 June 2015 (UTC)		:OpenZIM sounds like a good idea. What about providing a torrent too? With a RSS torrent feed, we could even update it automatically. --[[User:Quantum Immortal\|Quantum Immortal]] ([[User talk:Quantum Immortal\|talk]]) 06:18, 14 June 2015 (UTC)

−	Oooh, thank you very much! Now I can throw away that ugly wiki scraper I had lying around. :D 07:36, 19 June 2015 (UTC)	+	Oooh, thank you very much! Now I can throw away that ugly wiki scraper I had lying around. :D ——[[User:KeyboardFire\|KeyboardFire]] ([[User talk:KeyboardFire\|talk]]) 07:37, 19 June 2015 (UTC)

← Older revision		Revision as of 07:36, 19 June 2015
Line 18:		Line 18:

	:OpenZIM sounds like a good idea. What about providing a torrent too? With a RSS torrent feed, we could even update it automatically. --[[User:Quantum Immortal\|Quantum Immortal]] ([[User talk:Quantum Immortal\|talk]]) 06:18, 14 June 2015 (UTC)		:OpenZIM sounds like a good idea. What about providing a torrent too? With a RSS torrent feed, we could even update it automatically. --[[User:Quantum Immortal\|Quantum Immortal]] ([[User talk:Quantum Immortal\|talk]]) 06:18, 14 June 2015 (UTC)
		+
		+	Oooh, thank you very much! Now I can throw away that ugly wiki scraper I had lying around. :D 07:36, 19 June 2015 (UTC)

← Older revision		Revision as of 06:18, 14 June 2015
Line 16:		Line 16:

	Hi paxed, Is it possible to provide an openZIM formatted copy of NetHackWiki in addition to the XML dump? It would eliminate the headache of installing LAMP and mediawiki. Thanks. --[[User:Deadagain\|Deadagain]] ([[User talk:Deadagain\|talk]]) 13:49, 12 June 2015 (UTC)		Hi paxed, Is it possible to provide an openZIM formatted copy of NetHackWiki in addition to the XML dump? It would eliminate the headache of installing LAMP and mediawiki. Thanks. --[[User:Deadagain\|Deadagain]] ([[User talk:Deadagain\|talk]]) 13:49, 12 June 2015 (UTC)
		+
		+	:OpenZIM sounds like a good idea. What about providing a torrent too? With a RSS torrent feed, we could even update it automatically. --[[User:Quantum Immortal\|Quantum Immortal]] ([[User talk:Quantum Immortal\|talk]]) 06:18, 14 June 2015 (UTC)

← Older revision		Revision as of 13:49, 12 June 2015
Line 13:		Line 13:
	: You can try feeding the dump to a Perl script using modules [http://search.cpan.org/~triddle/MediaWiki-DumpFile-0.2.2/lib/MediaWiki/DumpFile/Pages.pm] and [http://search.cpan.org/~szabgab/Text-MediawikiFormat-1.04/lib/Text/MediawikiFormat.pm], thus transforming a giant XML into a giant HTML or a series of small HTMLs. [[Special:Contributions/95.165.173.36\|95.165.173.36]] 15:38, 27 May 2015 (UTC)		: You can try feeding the dump to a Perl script using modules [http://search.cpan.org/~triddle/MediaWiki-DumpFile-0.2.2/lib/MediaWiki/DumpFile/Pages.pm] and [http://search.cpan.org/~szabgab/Text-MediawikiFormat-1.04/lib/Text/MediawikiFormat.pm], thus transforming a giant XML into a giant HTML or a series of small HTMLs. [[Special:Contributions/95.165.173.36\|95.165.173.36]] 15:38, 27 May 2015 (UTC)
	: http://zdoom.org/wiki/Downloading_the_ZDoom_Wiki. [[User:ZzZombo\|ZzZombo]] ([[User talk:ZzZombo\|talk]]) 14:44, 4 June 2015 (UTC)		: http://zdoom.org/wiki/Downloading_the_ZDoom_Wiki. [[User:ZzZombo\|ZzZombo]] ([[User talk:ZzZombo\|talk]]) 14:44, 4 June 2015 (UTC)
		+
		+
		+	Hi paxed, Is it possible to provide an openZIM formatted copy of NetHackWiki in addition to the XML dump? It would eliminate the headache of installing LAMP and mediawiki. Thanks. --[[User:Deadagain\|Deadagain]] ([[User talk:Deadagain\|talk]]) 13:49, 12 June 2015 (UTC)