Author Topic: Farewell to Ornery  (Read 4631 times)

Kelcimer

  • Members
    • View Profile
Re: Farewell to Ornery
« Reply #50 on: August 31, 2021, 08:55:22 PM »
If you're able to rip them I'd be glad to host them along with the new forums. Just some flat files that have good inter-linking would do the trick.

My friend tells me that there is a way to create copy of the Old Forums that would not be dependent upon a database. It just need the Old Forums online to copy it. If the Cards could be patent for a week or so that would be great. I need to install some software and the start copying the Old Forums in sections. I am working through system capability issues with my computer to run the software, but hope to be able to try his solution by the weekend. If successful then I'll be happy to send it to you.

alai

  • Members
    • View Profile
Re: Farewell to Ornery
« Reply #51 on: August 31, 2021, 09:22:08 PM »
Web-crawling works great in the case where the database and the software works well enough to serve the pages, but you don't want to depend on it doing it for too much longer (or even if you quite understand exactly how it's doing it -- *makes sign of COBOL warding*).  If the software is broken, the better solution may be the offline port that Joshua describes, especially as backends are often standardised to a reasonable extent.  If the database itself if significantly corrupted, that's the point where people start sucking their teeth.  ("Just can't get the parts any more."  "Should have come to me last week!")

Kelcimer

  • Members
    • View Profile
Re: Farewell to Ornery
« Reply #52 on: September 05, 2021, 03:20:02 PM »
The cloning of the Old Forums is running.

I don't know if this will producing anything better than what JoshuaD can do with the the database for it, but it will be there as an option. Better to have it and not need it, then to need it and not have it, as it were.

The process is running slowly. There are a lot of files. It is presently 1/5th of the way through and has been running for almost two days. So if the forums can stay up through next weekend, that would be great.

JoshuaD

  • Members
    • View Profile
Re: Farewell to Ornery
« Reply #53 on: September 05, 2021, 09:25:48 PM »
That's a really long time, I wonder why that's happening.

We're going to be up here for a while longer.  I'm still waiting for the files for the new forums. I spoke with the current admin last week and he said he'd have it to me soonish.

Kelcimer

  • Members
    • View Profile
Re: Farewell to Ornery
« Reply #54 on: September 05, 2021, 11:31:47 PM »
Hello Joshua!

I wonder that as well. The program is supposed to follow every link on the forum and there are a lot of links on every page. Every member name is a link, and there is the "logged' link on each comment. Also, there are so many interconnected links. Like on a five page thread there will be the links to the other four pages at both the top and bottom of each page. So that is eight links per page on a five page thread and all of them are being checked. I understand that it makes a page for every link, so as that multiple pages have links to the same page, that winds up creating duplicate pages which it will de-duplicate after it has downloaded everything. And the forum ran for 15 years and there was a lot of interconnectedness. Maybe the fact that the Old Forums are still unstable is a factor as well.

Good to hear that we have more time.  :D

Greg Davidson

  • Members
    • View Profile
Re: Farewell to Ornery
« Reply #55 on: September 06, 2021, 10:42:33 PM »
My thanks to the Cards, and the rest of you here. I did listen and think about what each of you said, even those who were fervently arguing positions different from my own.

Stay well, and enjoy all the remaining days of your lives.

Fenring

  • Members
    • View Profile
Re: Farewell to Ornery
« Reply #56 on: September 06, 2021, 10:56:50 PM »
My thanks to the Cards, and the rest of you here. I did listen and think about what each of you said, even those who were fervently arguing positions different from my own.

Stay well, and enjoy all the remaining days of your lives.

How's the book coming?

Greg Davidson

  • Members
    • View Profile
Re: Farewell to Ornery
« Reply #57 on: September 07, 2021, 02:33:01 PM »
I started working with the acquisitions editor who originally bought and published The Red Tent for St. Martins Press (The Red Tent being perhaps the exemplar of successful mainstream biblical fiction). Her guidance was credible but pessimistic: the genre expectations for mainstream biblical fiction are to take an obscure female character and give her life while emphasizing themes of sisterhood and women's empowerment. That's what the target audience is looking for, and thus that's what publishers are looking for.

That's a reasonable expectation, and I could even envision such a novel (I'd love to write a novel about Jocheved, the last of the long-lived people in the bible and the only woman to be described as living for centuries). Next novel I will definitely think of the target market before starting! As for what to do with Book of Shem, I'm torn: part of me wants to finish the book the way I want, even if I need to self-publish, just so it can exist in the world (just this week I started a draft in the first person, to clarify and strengthen Shem's emotional depth). 

Part of me also wants to start something fresh with sales in mind. A friend of mine, a former VP of Avionics at Space-X, is finishing his first novel and he has a killer query letter (a launch vehicle is stolen in flight...). I have a pretty deep background with the James Webb Space Telescope (I was the program manager a NASA HQ for the first Hubble replacement instruments and later the Chandra X-ray Observatory, and when I started working for TRW it was to pull together a team to win that contract - I was even the deputy program manager for the job for a few years) and it's striking me that with launch coming up in the next few months, that background would have some cachet for a science fiction novel. But the innate contrarian in me would only get excited about a plot that shows how it is to actually build something like JWST (I loved Andy Weir's latest, but all science fiction authors seem clueless about how large organizations actually work to solve difficult challenges)

Fenring

  • Members
    • View Profile
Re: Farewell to Ornery
« Reply #58 on: September 07, 2021, 03:37:20 PM »
I'm torn: part of me wants to finish the book the way I want, even if I need to self-publish, just so it can exist in the world
[...]
Part of me also wants to start something fresh with sales in mind. A friend of mine, a former VP of Avionics at Space-X, is finishing his first novel and he has a killer query letter (a launch vehicle is

For what it's worth, I've also experienced the dilemma between thinking of what will be commercially attractive, compared to what your artistic instinct tell you needs to be done. In my case, in the theatre. There are two sides to it - a prisoner's (of the public) dilemma, of sorts. Either you can do what you felt compelled to start, knowing it might amount to nothing, but also knowing you did it for you, not for others (always a half-truth); but with the chance that when something of quality comes into existence it can create its own market. This is what happens whenever something new comes along that happens to end up successful. And the other side of it is to avoid wasting time on non-viable projects, and to create something to fill an apparent niche, and hope to grow into it spiritually; but with the chance that even if you follow a template to maximize commercial success it may still fall flat on its face. And at that point, why care at all what the public wanted after all? So that's a tough one. Most artistic projects don't become commercial successes, and it's only correlated slightly to their merit.

My very personal tendency would be to do the thing I feel needs doing, not what the public will receive best. But then again, I'm not famous. So don't take this as career advice, more like, a slice of an Ornery poster's life.

Fenring

  • Members
    • View Profile
Re: Farewell to Ornery
« Reply #59 on: September 07, 2021, 03:45:40 PM »
the genre expectations for mainstream biblical fiction are to take an obscure female character and give her life while emphasizing themes of sisterhood and women's empowerment. That's what the target audience is looking for, and thus that's what publishers are looking for.

Since I realize my last post was mostly about me, rather than about you, I thought I'd throw this in: if stories about female empowerment and sisterhood were what you desperately wanted to write about, why wouldn't you already be doing so? I know plenty of people in the arts passionate about this subject and (no offense intended) no one needed to suggest to them to write stories about that; they can hardly stop themselves finding those themes in every piece they encounter. So I would think long and hard about whether it's ideal (or even decent?) to take on a topic that doesn't already speak deeply to you, that you can speak with some genuine intuitions about (as you suggest about cooperative engineering in sci-fi). I don't know if the feminine empowerment theme does or doesn't fill this bill for you, but what I always ask artists is: why should we see you perform, or read your writing? Something about it has to be uniquely something you could give that no one else could have. So what is that for you? (rhetorical) We want Greg Davidson, not someone else.

cherrypoptart

  • Members
    • View Profile
Re: Farewell to Ornery
« Reply #60 on: September 07, 2021, 06:57:43 PM »
I've been holding off on saying this since I didn't want to jinx it but it's looking solid enough now.

Re: Farewell to Ornery

“Good news. The reports of my death are greatly exaggerated." - Ornery.org

Greg Davidson

  • Members
    • View Profile
Re: Farewell to Ornery
« Reply #61 on: September 08, 2021, 12:27:37 PM »
Thanks, Fenrig.

I think I could "yes, and..." this - finish off Book of Shem in a way that satisfies me (for example, I've been targeting a trilogy of 80K word novels and selling the first as a standalone because that's the length expectation for a first time author, but it's really one story that I could probably tell in about 180K words). But at the same time, pick a topic that both energizes me and has a recognizable genre home.

Seriati

  • Members
    • View Profile
Re: Farewell to Ornery
« Reply #62 on: September 08, 2021, 05:46:28 PM »
Her guidance was credible but pessimistic: the genre expectations for mainstream biblical fiction are to take an obscure female character and give her life while emphasizing themes of sisterhood and women's empowerment. That's what the target audience is looking for, and thus that's what publishers are looking for.

You may want to consider trying a publisher in a different genre.  Recast it as a fantasy based on a biblical story and you could pitch it through a fantasy publisher to an entirely different target market.  I wouldn't be surprised at all to see a book like that in a Fantasy section and never once walked into  biblical fiction section.

Quote
A friend of mine, a former VP of Avionics at Space-X, is finishing his first novel and he has a killer query letter (a launch vehicle is stolen in flight...).

Isn't that the literal plot of the James Bond movie Moonraker?  ;)

Quote
...but all science fiction authors seem clueless about how large organizations actually work to solve difficult challenges)

That actually applies to all fiction authors, and a good number of non-fiction authors as well.  It's amazing how often the author's own contributions to a project were the "key" without which it all would have failed. 

Fenring

  • Members
    • View Profile
Re: Farewell to Ornery
« Reply #63 on: September 08, 2021, 11:12:25 PM »
But at the same time, pick a topic that both energizes me and has a recognizable genre home.

I hope you can find that happy marriage. I've done some writing (theatre material and non-fiction for teaching purposes), never attempted a novel, but I understand that pushing through a project that big can be like torture at times. I imagine it helps a lot if it's something you've got to write. In my case, some of my bigger writing/editing projects only got completed in due time because I had hard deadlines that were immovable.

TheDeamon

  • Members
    • View Profile
Re: Farewell to Ornery
« Reply #64 on: September 10, 2021, 09:45:56 AM »
the genre expectations for mainstream biblical fiction are to take an obscure female character and give her life while emphasizing themes of sisterhood and women's empowerment. That's what the target audience is looking for, and thus that's what publishers are looking for.

Since I realize my last post was mostly about me, rather than about you, I thought I'd throw this in: if stories about female empowerment and sisterhood were what you desperately wanted to write about, why wouldn't you already be doing so? I know plenty of people in the arts passionate about this subject and (no offense intended) no one needed to suggest to them to write stories about that; they can hardly stop themselves finding those themes in every piece they encounter. So I would think long and hard about whether it's ideal (or even decent?) to take on a topic that doesn't already speak deeply to you, that you can speak with some genuine intuitions about (as you suggest about cooperative engineering in sci-fi). I don't know if the feminine empowerment theme does or doesn't fill this bill for you, but what I always ask artists is: why should we see you perform, or read your writing? Something about it has to be uniquely something you could give that no one else could have. So what is that for you? (rhetorical) We want Greg Davidson, not someone else.

The thing that get caught up in the process about those "passionate about female empowerment" is that many of them end up delving too deep. Either they end up with something only others who are "equally passionate" on the subject matter can begin to decipher(a minor in "women's studies" shouldn't be required), or they simply end up writing screeds that send many readers running for any of a number of reasons.

Kelcimer

  • Members
    • View Profile
Re: Farewell to Ornery
« Reply #65 on: September 19, 2021, 12:48:10 AM »
Just to say, the web crawl is still proceeding. There are a lot of links and it is proceeding slowly.

JoshuaD

  • Members
    • View Profile
Re: Farewell to Ornery
« Reply #66 on: September 24, 2021, 03:36:24 PM »
Just to say, the web crawl is still proceeding. There are a lot of links and it is proceeding slowly.

I am worried that you are ripping the entire internet.  There's no way it should take 14 days rip a single forum.

TheDrake

  • Members
    • View Profile
Re: Farewell to Ornery
« Reply #67 on: September 24, 2021, 03:42:46 PM »
Just to say, the web crawl is still proceeding. There are a lot of links and it is proceeding slowly.

I am worried that you are ripping the entire internet.  There's no way it should take 14 days rip a single forum.

Just speculating, it could easily take that long if it renders each page by independent database access and it is single threaded.

In my experience as a technical manager, anyone who claims something "should never take that long" is mistaken about the scope of the task or the resources available.

LetterRip

  • Members
    • View Profile
Re: Farewell to Ornery
« Reply #68 on: September 24, 2021, 04:07:35 PM »
What crawler are you using?

Is it traversing via a link traversal search (pick a start page, gathers all links on page, traverses each link)
Or page index increment?

for instance the current ornery pages are formatted as

Code: [Select]
http://www.ornery.org/forum/index.php/topic,TOPIC_ID.PG_NUM.html
where TOPIC_ID starts at 6 and increments per topic post; and PG_NUM is 1, 50, 100, 150, 200, 250, etc with the 50 incremented for each new page.

so would be pretty trivial to index.

Not sure what the old forum indexing was setup as.

Looks like the other forum is

Code: [Select]
http://www.ornery.org/cgi-bin/ubbcgi/ultimatebb.cgi/topic/6/TOPIC_ID.html
and

Code: [Select]
http://www.ornery.org/cgi-bin/ubbcgi/ultimatebb.cgi/topic/6/TOPIC_ID/PAGE_ID.html
TOPIC_ID starts at 1321 (didn't check all numbers below that)
there are some discontinuties - but goes up to about 17000.

If the first page, then uses the first format with just the TOPIC_ID, else PAGE_ID is 2,3,4,5 etc.

« Last Edit: September 24, 2021, 04:20:54 PM by LetterRip »

LetterRip

  • Members
    • View Profile
Re: Farewell to Ornery
« Reply #69 on: September 24, 2021, 04:27:19 PM »
Assuming average number of 2 pages per topic, and 3 seconds per page (to avoid throttling) that would take 27 hours for all pages (mean of 2 pages might seem low, but most of our history 80% of topics only had a single page of replies).

LetterRip

  • Members
    • View Profile
Re: Farewell to Ornery
« Reply #70 on: September 24, 2021, 04:30:12 PM »
The archives also have their own set of archives but they are much smaller.  7 pages of topics (2002-2006) but the main archive is 2002-2015 (383 pages of topics).  Makes me curious if they cover the same content.
« Last Edit: September 24, 2021, 04:33:17 PM by LetterRip »

Kelcimer

  • Members
    • View Profile
Re: Farewell to Ornery
« Reply #71 on: September 30, 2021, 08:40:27 PM »
I wasn't able to get it to work on my old computer (it is a decade old). My friend was kind enough to run it on his computer.

There are about 615,000 comments, and if each button on a comment counts as a link that needs to be checked, then that is 6 links per comment. That would mean checking about about 3.6 million links. A back of the envelope calculation for every other link on each page would bring up total number of links that need to be checked to about 4 million. Does that sound about right?

I don't know why it should take this long, but my friend indicates the program is still chugging away.

Kelcimer

  • Members
    • View Profile
Re: Farewell to Ornery
« Reply #72 on: October 14, 2021, 10:57:00 PM »
Good news/stupid news

Good news: I opened the clone website in a browser and looking around for a while. It looks like it is all there. So if they were holding off on tranfering the website or anything because of this, they need hold off no longer.

Stupid news: The file size of the whole thing is insanely huge. Compressed it is 4gigs, uncompressed it is about 50 gigs. I was trying to figure out why it was so large and it looks like it duplicated every page every couple of days. So it technically isn't a copy of the archives, but many, many copies of the archives. I made the mistake of opening a folder window to look at the pages to figure out why it was so large. Turns out Finder on a ten year old mac is not up to rendering the absurd number of duplicated file names in one window. Finder got hot and was acting weird so I did a shut down/restart and my computer did not want to restart.

When I get my new computer I plan to see about deleting the dupes, get it to a reasonable file size, and make the clone of the archives available to JoshuaD.


cherrypoptart

  • Members
    • View Profile
Re: Farewell to Ornery
« Reply #73 on: October 15, 2021, 05:32:05 AM »
I've seen some people if they want to share files with everyone and anyone, they put it in a Google drive on kind of a throwaway account. It looks like you get 15 gigs free so I wonder if that could be an option to just let all of us and anyone else who wants it have a copy.

LetterRip

  • Members
    • View Profile
Re: Farewell to Ornery
« Reply #74 on: October 15, 2021, 01:23:46 PM »
Writing an algorithm that dedups them probably wouldn't be hard.

if you run this bit of python, you can just send me the file created.  This assumes they are all html files in folders, if they are zipped, or in a database or something, or use a different file type, will need more information...

Code: [Select]
#paste into a text file and name it - list_files.py
#then replace path and out_folder_path with valid paths inside the quotation marks
#run by typing python list_files.py on a terminal prompt
import os
path = r"/path/to/root/folder_name" #navigate to the folder that contains the folder then copy and replace the file path in this file.
out_folder_path = r"/path/to/write/files" #replace this path with the folder you want the file list_of_fnames.txt written to
#then email me the list_of_fnames.txt

path_list = []
for folder, subfolder, fnames in os.walk(path):
  for fname in fnames:
    if fname.endswith(".html"):
      path_list.append(folder+os.sep+fname)
  with open(out_folder_path+os.sep+"list_of_fnames.txt", "w") as f:
    f.write("\n".join(path_list)
 

Kelcimer

  • Members
    • View Profile
Re: Farewell to Ornery
« Reply #75 on: October 15, 2021, 05:27:55 PM »
I've seen some people if they want to share files with everyone and anyone, they put it in a Google drive on kind of a throwaway account. It looks like you get 15 gigs free so I wonder if that could be an option to just let all of us and anyone else who wants it have a copy.

I like that idea.

LetterRip,

It appeared that the dublicate file names were like so:

filename.html
filename-0002.html
filename-0003.html
filename-0004.html
etc....

LetterRip

  • Members
    • View Profile
Re: Farewell to Ornery
« Reply #76 on: October 16, 2021, 07:29:36 AM »
LetterRip,

It appeared that the dublicate file names were like so:

filename.html
filename-0002.html
filename-0003.html
filename-0004.html
etc....

Have you checked that the 0002, 0003 etc aren't the 'page 2' and page 3' etc?

Kelcimer

  • Members
    • View Profile
Re: Farewell to Ornery
« Reply #77 on: October 16, 2021, 02:23:52 PM »
I don't think that is the case. I wasn't able to confirm by directly looking at the files, because my mac didn't function that long. I want to say it went up to about 0036 on every file that I could see. We didn't have many threads go as high as 30 pages.

Kelcimer

  • Members
    • View Profile
Re: Farewell to Ornery
« Reply #78 on: November 07, 2021, 01:52:00 AM »
Okay. I have my new computer and attempted to look at the files in the folder to see what was going on.

The way the file names went were like so:

Quote
ultimatebb.html
ultimatebb0000-2.html
ultimatebb0000-3.html
ultimatebb0000-4.html
ultimatebb0000-5.html
ultimatebb0000-6.html
ultimatebb0000-7.html

[...]

ultimatebb0000-37.html
ultimatebb0000-38.html
ultimatebb0000-39.html
ultimatebb0000-40.html
ultimatebb0000-41.html
ultimatebb0000-42.html
ultimatebb0000-43.html

Of those 43 pages, the original and #37 were actual pages. The rest were 10KB and say "You are not logged in. You must be logged in (and registered) to perform this function"

It looks like it kept returning to each page to check those links again and again.

I was looking around in the folder and the thing locked up my new Mac Mini and it became hot to the touch. I had to shut down my brand new computer because of what it was doing. No wonder it took out my old computer.

Trying to figure out how to cut down the the overall file size is not readily apparent to me. I want to share the clone with anyone who wants it, but I wouldn't want this version to be the one shared. It needs to loose a lot of file size.

Anybody have any advice? Anybody want to take a direct crack at it?

LetterRip

  • Members
    • View Profile
Re: Farewell to Ornery
« Reply #79 on: November 07, 2021, 09:23:24 AM »
Email me two of the pages with the bad information and one good page, and I'll send you a script that delete all of the bad pages.

Are the originals still up?
« Last Edit: November 07, 2021, 09:26:13 AM by LetterRip »

LetterRip

  • Members
    • View Profile
Re: Farewell to Ornery
« Reply #80 on: November 07, 2021, 10:08:40 AM »
Hmm looks like the originals are not up anymore...  So hopefully most of the content was scraped...

Kelcimer

  • Members
    • View Profile
Re: Farewell to Ornery
« Reply #81 on: November 08, 2021, 12:42:30 AM »
I sent you an email.

At about 50 gigs, I should hope that all pages were got at least once.

There were some pages that were of a single post with a link to the page that the post appeared on. I don't know how much to make of that. Which is to say there may be more oddities that need addressing.

That said, removing all of the "You are not logged in" pages would be a good start.

Thank you!

Kelcimer

  • Members
    • View Profile
Re: Farewell to Ornery
« Reply #82 on: November 11, 2021, 01:42:05 PM »
LetterRip.

For clarity, I sent an email through the website. I do not know with the change over if that function still works.

It does not allow attachments so I included my email address. Let me know if you received it.

John

LetterRip

  • Members
    • View Profile
Re: Farewell to Ornery
« Reply #83 on: November 11, 2021, 02:29:53 PM »
It worked, replied via email...