Farewell to Ornery

[Kelcimer]

If you're able to rip them I'd be glad to host them along with the new forums. Just some flat files that have good inter-linking would do the trick.

My friend tells me that there is a way to create copy of the Old Forums that would not be dependent upon a database. It just need the Old Forums online to copy it. If the Cards could be patent for a week or so that would be great. I need to install some software and the start copying the Old Forums in sections. I am working through system capability issues with my computer to run the software, but hope to be able to try his solution by the weekend. If successful then I'll be happy to send it to you.

[alai]

Web-crawling works great in the case where the database and the software works well enough to serve the pages, but you don't want to depend on it doing it for too much longer (or even if you quite understand exactly how it's doing it -- *makes sign of COBOL warding*).  If the software is broken, the better solution may be the offline port that Joshua describes, especially as backends are often standardised to a reasonable extent.  If the database itself if significantly corrupted, that's the point where people start sucking their teeth.  ("Just can't get the parts any more."  "Should have come to me last week!")

[Kelcimer]

The cloning of the Old Forums is running.

I don't know if this will producing anything better than what JoshuaD can do with the the database for it, but it will be there as an option. Better to have it and not need it, then to need it and not have it, as it were.

The process is running slowly. There are a lot of files. It is presently 1/5th of the way through and has been running for almost two days. So if the forums can stay up through next weekend, that would be great.

[JoshuaD]

That's a really long time, I wonder why that's happening.

We're going to be up here for a while longer.  I'm still waiting for the files for the new forums. I spoke with the current admin last week and he said he'd have it to me soonish.

[Kelcimer]

Hello Joshua!

I wonder that as well. The program is supposed to follow every link on the forum and there are a lot of links on every page. Every member name is a link, and there is the "logged' link on each comment. Also, there are so many interconnected links. Like on a five page thread there will be the links to the other four pages at both the top and bottom of each page. So that is eight links per page on a five page thread and all of them are being checked. I understand that it makes a page for every link, so as that multiple pages have links to the same page, that winds up creating duplicate pages which it will de-duplicate after it has downloaded everything. And the forum ran for 15 years and there was a lot of interconnectedness. Maybe the fact that the Old Forums are still unstable is a factor as well.

Good to hear that we have more time. 

[Greg Davidson]

My thanks to the Cards, and the rest of you here. I did listen and think about what each of you said, even those who were fervently arguing positions different from my own.

Stay well, and enjoy all the remaining days of your lives.

[Fenring]

My thanks to the Cards, and the rest of you here. I did listen and think about what each of you said, even those who were fervently arguing positions different from my own.

Stay well, and enjoy all the remaining days of your lives.

How's the book coming?

[Greg Davidson]

I started working with the acquisitions editor who originally bought and published The Red Tent for St. Martins Press (The Red Tent being perhaps the exemplar of successful mainstream biblical fiction). Her guidance was credible but pessimistic: the genre expectations for mainstream biblical fiction are to take an obscure female character and give her life while emphasizing themes of sisterhood and women's empowerment. That's what the target audience is looking for, and thus that's what publishers are looking for.

That's a reasonable expectation, and I could even envision such a novel (I'd love to write a novel about Jocheved, the last of the long-lived people in the bible and the only woman to be described as living for centuries). Next novel I will definitely think of the target market before starting! As for what to do with Book of Shem, I'm torn: part of me wants to finish the book the way I want, even if I need to self-publish, just so it can exist in the world (just this week I started a draft in the first person, to clarify and strengthen Shem's emotional depth). 

Part of me also wants to start something fresh with sales in mind. A friend of mine, a former VP of Avionics at Space-X, is finishing his first novel and he has a killer query letter (a launch vehicle is stolen in flight...). I have a pretty deep background with the James Webb Space Telescope (I was the program manager a NASA HQ for the first Hubble replacement instruments and later the Chandra X-ray Observatory, and when I started working for TRW it was to pull together a team to win that contract - I was even the deputy program manager for the job for a few years) and it's striking me that with launch coming up in the next few months, that background would have some cachet for a science fiction novel. But the innate contrarian in me would only get excited about a plot that shows how it is to actually build something like JWST (I loved Andy Weir's latest, but all science fiction authors seem clueless about how large organizations actually work to solve difficult challenges)

[Fenring]

I'm torn: part of me wants to finish the book the way I want, even if I need to self-publish, just so it can exist in the world
[...]
Part of me also wants to start something fresh with sales in mind. A friend of mine, a former VP of Avionics at Space-X, is finishing his first novel and he has a killer query letter (a launch vehicle is

For what it's worth, I've also experienced the dilemma between thinking of what will be commercially attractive, compared to what your artistic instinct tell you needs to be done. In my case, in the theatre. There are two sides to it - a prisoner's (of the public) dilemma, of sorts. Either you can do what you felt compelled to start, knowing it might amount to nothing, but also knowing you did it for you, not for others (always a half-truth); but with the chance that when something of quality comes into existence it can create its own market. This is what happens whenever something new comes along that happens to end up successful. And the other side of it is to avoid wasting time on non-viable projects, and to create something to fill an apparent niche, and hope to grow into it spiritually; but with the chance that even if you follow a template to maximize commercial success it may still fall flat on its face. And at that point, why care at all what the public wanted after all? So that's a tough one. Most artistic projects don't become commercial successes, and it's only correlated slightly to their merit.

My very personal tendency would be to do the thing I feel needs doing, not what the public will receive best. But then again, I'm not famous. So don't take this as career advice, more like, a slice of an Ornery poster's life.

[Fenring]

the genre expectations for mainstream biblical fiction are to take an obscure female character and give her life while emphasizing themes of sisterhood and women's empowerment. That's what the target audience is looking for, and thus that's what publishers are looking for.

Since I realize my last post was mostly about me, rather than about you, I thought I'd throw this in: if stories about female empowerment and sisterhood were what you desperately wanted to write about, why wouldn't you already be doing so? I know plenty of people in the arts passionate about this subject and (no offense intended) no one needed to suggest to them to write stories about that; they can hardly stop themselves finding those themes in every piece they encounter. So I would think long and hard about whether it's ideal (or even decent?) to take on a topic that doesn't already speak deeply to you, that you can speak with some genuine intuitions about (as you suggest about cooperative engineering in sci-fi). I don't know if the feminine empowerment theme does or doesn't fill this bill for you, but what I always ask artists is: why should we see you perform, or read your writing? Something about it has to be uniquely something you could give that no one else could have. So what is that for you? (rhetorical) We want Greg Davidson, not someone else.

[cherrypoptart]

I've been holding off on saying this since I didn't want to jinx it but it's looking solid enough now.

Re: Farewell to Ornery

“Good news. The reports of my death are greatly exaggerated." - Ornery.org

[Greg Davidson]

Thanks, Fenrig.

I think I could "yes, and..." this - finish off Book of Shem in a way that satisfies me (for example, I've been targeting a trilogy of 80K word novels and selling the first as a standalone because that's the length expectation for a first time author, but it's really one story that I could probably tell in about 180K words). But at the same time, pick a topic that both energizes me and has a recognizable genre home.

[Seriati]

Her guidance was credible but pessimistic: the genre expectations for mainstream biblical fiction are to take an obscure female character and give her life while emphasizing themes of sisterhood and women's empowerment. That's what the target audience is looking for, and thus that's what publishers are looking for.

You may want to consider trying a publisher in a different genre.  Recast it as a fantasy based on a biblical story and you could pitch it through a fantasy publisher to an entirely different target market.  I wouldn't be surprised at all to see a book like that in a Fantasy section and never once walked into  biblical fiction section.

A friend of mine, a former VP of Avionics at Space-X, is finishing his first novel and he has a killer query letter (a launch vehicle is stolen in flight...).

Isn't that the literal plot of the James Bond movie Moonraker? 

...but all science fiction authors seem clueless about how large organizations actually work to solve difficult challenges)

That actually applies to all fiction authors, and a good number of non-fiction authors as well.  It's amazing how often the author's own contributions to a project were the "key" without which it all would have failed. 

[Fenring]

But at the same time, pick a topic that both energizes me and has a recognizable genre home.

I hope you can find that happy marriage. I've done some writing (theatre material and non-fiction for teaching purposes), never attempted a novel, but I understand that pushing through a project that big can be like torture at times. I imagine it helps a lot if it's something you've got to write. In my case, some of my bigger writing/editing projects only got completed in due time because I had hard deadlines that were immovable.

[TheDeamon]

the genre expectations for mainstream biblical fiction are to take an obscure female character and give her life while emphasizing themes of sisterhood and women's empowerment. That's what the target audience is looking for, and thus that's what publishers are looking for.

Since I realize my last post was mostly about me, rather than about you, I thought I'd throw this in: if stories about female empowerment and sisterhood were what you desperately wanted to write about, why wouldn't you already be doing so? I know plenty of people in the arts passionate about this subject and (no offense intended) no one needed to suggest to them to write stories about that; they can hardly stop themselves finding those themes in every piece they encounter. So I would think long and hard about whether it's ideal (or even decent?) to take on a topic that doesn't already speak deeply to you, that you can speak with some genuine intuitions about (as you suggest about cooperative engineering in sci-fi). I don't know if the feminine empowerment theme does or doesn't fill this bill for you, but what I always ask artists is: why should we see you perform, or read your writing? Something about it has to be uniquely something you could give that no one else could have. So what is that for you? (rhetorical) We want Greg Davidson, not someone else.

The thing that get caught up in the process about those "passionate about female empowerment" is that many of them end up delving too deep. Either they end up with something only others who are "equally passionate" on the subject matter can begin to decipher(a minor in "women's studies" shouldn't be required), or they simply end up writing screeds that send many readers running for any of a number of reasons.

[Kelcimer]

Just to say, the web crawl is still proceeding. There are a lot of links and it is proceeding slowly.

[JoshuaD]

Just to say, the web crawl is still proceeding. There are a lot of links and it is proceeding slowly.

I am worried that you are ripping the entire internet.  There's no way it should take 14 days rip a single forum.

[TheDrake]

Just to say, the web crawl is still proceeding. There are a lot of links and it is proceeding slowly.

I am worried that you are ripping the entire internet.  There's no way it should take 14 days rip a single forum.

Just speculating, it could easily take that long if it renders each page by independent database access and it is single threaded.

In my experience as a technical manager, anyone who claims something "should never take that long" is mistaken about the scope of the task or the resources available.

[LetterRip]

What crawler are you using?

Is it traversing via a link traversal search (pick a start page, gathers all links on page, traverses each link)
Or page index increment?

for instance the current ornery pages are formatted as

Code: [Select]

http://www.ornery.org/forum/index.php/topic,TOPIC_ID.PG_NUM.html
where TOPIC_ID starts at 6 and increments per topic post; and PG_NUM is 1, 50, 100, 150, 200, 250, etc with the 50 incremented for each new page.

so would be pretty trivial to index.

Not sure what the old forum indexing was setup as.

Looks like the other forum is

Code: [Select]

http://www.ornery.org/cgi-bin/ubbcgi/ultimatebb.cgi/topic/6/TOPIC_ID.html
and

Code: [Select]

http://www.ornery.org/cgi-bin/ubbcgi/ultimatebb.cgi/topic/6/TOPIC_ID/PAGE_ID.html
TOPIC_ID starts at 1321 (didn't check all numbers below that)
there are some discontinuties - but goes up to about 17000.

If the first page, then uses the first format with just the TOPIC_ID, else PAGE_ID is 2,3,4,5 etc.

[LetterRip]

Assuming average number of 2 pages per topic, and 3 seconds per page (to avoid throttling) that would take 27 hours for all pages (mean of 2 pages might seem low, but most of our history 80% of topics only had a single page of replies).

[LetterRip]

The archives also have their own set of archives but they are much smaller.  7 pages of topics (2002-2006) but the main archive is 2002-2015 (383 pages of topics).  Makes me curious if they cover the same content.

[Kelcimer]

I wasn't able to get it to work on my old computer (it is a decade old). My friend was kind enough to run it on his computer.

There are about 615,000 comments, and if each button on a comment counts as a link that needs to be checked, then that is 6 links per comment. That would mean checking about about 3.6 million links. A back of the envelope calculation for every other link on each page would bring up total number of links that need to be checked to about 4 million. Does that sound about right?

I don't know why it should take this long, but my friend indicates the program is still chugging away.

[Kelcimer]

Good news/stupid news

Good news: I opened the clone website in a browser and looking around for a while. It looks like it is all there. So if they were holding off on tranfering the website or anything because of this, they need hold off no longer.

Stupid news: The file size of the whole thing is insanely huge. Compressed it is 4gigs, uncompressed it is about 50 gigs. I was trying to figure out why it was so large and it looks like it duplicated every page every couple of days. So it technically isn't a copy of the archives, but many, many copies of the archives. I made the mistake of opening a folder window to look at the pages to figure out why it was so large. Turns out Finder on a ten year old mac is not up to rendering the absurd number of duplicated file names in one window. Finder got hot and was acting weird so I did a shut down/restart and my computer did not want to restart.

When I get my new computer I plan to see about deleting the dupes, get it to a reasonable file size, and make the clone of the archives available to JoshuaD.

[cherrypoptart]

I've seen some people if they want to share files with everyone and anyone, they put it in a Google drive on kind of a throwaway account. It looks like you get 15 gigs free so I wonder if that could be an option to just let all of us and anyone else who wants it have a copy.

[LetterRip]

Writing an algorithm that dedups them probably wouldn't be hard.

if you run this bit of python, you can just send me the file created.  This assumes they are all html files in folders, if they are zipped, or in a database or something, or use a different file type, will need more information...

Code: [Select]

#paste into a text file and name it - list_files.py
#then replace path and out_folder_path with valid paths inside the quotation marks
#run by typing python list_files.py on a terminal prompt
import os
path = r"/path/to/root/folder_name" #navigate to the folder that contains the folder then copy and replace the file path in this file.
out_folder_path = r"/path/to/write/files" #replace this path with the folder you want the file list_of_fnames.txt written to
#then email me the list_of_fnames.txt

path_list = []
for folder, subfolder, fnames in os.walk(path):
  for fname in fnames:
    if fname.endswith(".html"):
      path_list.append(folder+os.sep+fname)
  with open(out_folder_path+os.sep+"list_of_fnames.txt", "w") as f:
    f.write("\n".join(path_list)
 

[Kelcimer]

I've seen some people if they want to share files with everyone and anyone, they put it in a Google drive on kind of a throwaway account. It looks like you get 15 gigs free so I wonder if that could be an option to just let all of us and anyone else who wants it have a copy.

I like that idea.

LetterRip,

It appeared that the dublicate file names were like so:

filename.html
filename-0002.html
filename-0003.html
filename-0004.html
etc....

[LetterRip]

LetterRip,

It appeared that the dublicate file names were like so:

filename.html
filename-0002.html
filename-0003.html
filename-0004.html
etc....

Have you checked that the 0002, 0003 etc aren't the 'page 2' and page 3' etc?

[Kelcimer]

I don't think that is the case. I wasn't able to confirm by directly looking at the files, because my mac didn't function that long. I want to say it went up to about 0036 on every file that I could see. We didn't have many threads go as high as 30 pages.

[Kelcimer]

Okay. I have my new computer and attempted to look at the files in the folder to see what was going on.

The way the file names went were like so:

ultimatebb.html
ultimatebb0000-2.html
ultimatebb0000-3.html
ultimatebb0000-4.html
ultimatebb0000-5.html
ultimatebb0000-6.html
ultimatebb0000-7.html

[...]

ultimatebb0000-37.html
ultimatebb0000-38.html
ultimatebb0000-39.html
ultimatebb0000-40.html
ultimatebb0000-41.html
ultimatebb0000-42.html
ultimatebb0000-43.html

Of those 43 pages, the original and #37 were actual pages. The rest were 10KB and say "You are not logged in. You must be logged in (and registered) to perform this function"

It looks like it kept returning to each page to check those links again and again.

I was looking around in the folder and the thing locked up my new Mac Mini and it became hot to the touch. I had to shut down my brand new computer because of what it was doing. No wonder it took out my old computer.

Trying to figure out how to cut down the the overall file size is not readily apparent to me. I want to share the clone with anyone who wants it, but I wouldn't want this version to be the one shared. It needs to loose a lot of file size.

Anybody have any advice? Anybody want to take a direct crack at it?

[LetterRip]

Email me two of the pages with the bad information and one good page, and I'll send you a script that delete all of the bad pages.

Are the originals still up?

[LetterRip]

Hmm looks like the originals are not up anymore...  So hopefully most of the content was scraped...

[Kelcimer]

I sent you an email.

At about 50 gigs, I should hope that all pages were got at least once.

There were some pages that were of a single post with a link to the page that the post appeared on. I don't know how much to make of that. Which is to say there may be more oddities that need addressing.

That said, removing all of the "You are not logged in" pages would be a good start.

Thank you!

[Kelcimer]

LetterRip.

For clarity, I sent an email through the website. I do not know with the change over if that function still works.

It does not allow attachments so I included my email address. Let me know if you received it.

John

[LetterRip]

It worked, replied via email...

[fizz]

People, I'm saying goodbye to everybody.
I've never been a particularly active contributor, because English is not my native language and writing long essays in it is quite tiring for me, and with my work I honestly do not have much time to dedicate to this, but I've lurked around here for about 20 years by now.

I arrived here like may because I liked the first couple Ender novels, was a bit shocked by learning the real OSC politics, but still remained because it was for me an invaluable insight in the way of thinking on your side of the pond, like the British would say.
Also, I appreciated the relatively civil and and well argumented style of discourse: I'm sorry to say that most of my country forums tended more to the youtube comment section kind of name-calling, than true discussion.

I had almost abandoned it, when the need to try to understand the Trump phenomenon drove me back: it was such an utterly incomprehensible thing that i had to see from the.. keyboards.. of real americans what kind of thoughts passed in the mind of a Trump supporter.

But now, I admit I'm getting tired. I'm burned out, feel the need to argue when I read some things (https://xkcd.com/386/) but I'm at the same time getting more and more resigned to the futility of any attempt at discussion.
Having a novax conspiracy theorist old father also makes that feeling hit a bit too close to home.
It may be also age, work stress and/or pandemic burn out, but nowaday when I start discussing I take it... personally.. it stresses me out, it tend to make me dis proportionally angry: where years ago I would have enjoyed it (even if as I said i was unable to do it much here), now it sends me in a bad place.

I'm thankfully already not on any social network (no twitter, no facebook, no instagram or anything else), my only media presence is here and a couple of news sites comment sections.

Sooo, i'm publicly committing myself to stop coming here and commenting. 20 years ago when I started coming here I had just quit smoking, and it kept till now, i hope I will be as good as that with internet commentary too :-p

Goodbye, and thanks for all the fish! (cit.)

[cherrypoptart]

I enjoyed the additional European perspective. Take a break for as long as you like but just always know that there's a key under the mat and you're welcome back anytime for as long or as short of a time as you'd like.

On the bright side, it is something to be thankful for that the Farewell to Ornery thread can now be repurposed for people saying farewell, hopefully just for a while, as opposed to all of us having to say goodbye to Ornery forever.

[JoshuaD]

Goodbye Fizz. I hope you stay around to finish our conversation in the God thread, but I understand if not.

[JoshuaD]

FWIW: I often take long bouts of breaks from posting. It can be exhausting. But I come back for a thread here and there, and like having it for that much.

[rightleft22]

But now, I admit I'm getting tired. I'm burned out, feel the need to argue when I read some things but I'm at the same time getting more and more resigned to the futility of any attempt at discussion.

Curious how many feel this way.

I used to enjoy a good debate if mostly as a observer. Now what passes for debate leaves me feeling discouraged if not angry.

I'm not sure how we got to a place where our opinions had to matter and that they only matter if everyone agrees with how 'I' see things.

My opinions have never changed anything on the macro level, life let alone most others don't give a rat's ass what I think. I don't want to be the angry old man or crazy uncle so I let it go.   

Add to that, I had to accept that I'm a horrible communicator and seldom articulates my thoughts clearly. 

[TheDeamon]

For me, it shifted long ago. I largely don't engage in such discussions to change people's minds.

I engage in order to gain at least a modicum of understanding as to where they're coming from on their side of things. Especially with how big tech has tried to stuff all of us into information bubbles, asking questions of people in different bubbles does have a decent chance of revealing new information.

[JoshuaD]

Yeah, I've retreated from trying to change people's minds. I mostly post to crowd-source for criticisms of what I believe and to be forced to explicitly articulate some things. I am happy if that process is helpful for other people and I try to write to maximize the chance that it is, but it's not really my goal most of the time.

[Wayward Son]

I'm not sure how we got to a place where our opinions had to matter and that they only matter if everyone agrees with how 'I' see things.

I believe the problem is that the stakes have become higher.

Before, we just disagreed, but we all believed that most people followed the rules.  Majorities won, tempered by the courts, but there was always the next election to turn things around if necessary.  One could always expect things to go back to the norm.

Now it feels like it is winner-take-all.  That those in power will manipulate the system to make themselves permanently in power.  That they will re-write the rules, break them if necessary, or just destroy those who try to impose the rules on them.  It has become all-out war.

Those who support such people are an actual danger to everyone else.  They provide them cover and power, giving them the opportunity to solidify their power, to make it permanent.  So it's no longer just another opinion to win out in the arena of opinions.  It is a side that is trying to oppress and destroy all opposition.

Admittedly, some things are still opinions.   But it is increasingly becoming a war of sides.  And how can you tolerate the "other side" when it's goal is to destroy everything you believe? 

[alai]

“Good news. The reports of my death are greatly exaggerated." - Ornery.org

Twain, of course, wasn't so immodest as to claim the first part!