Bots scanning The Corner?

[DTEJD1997]

Hey all:

 

I've noticed something for quite a while that I never made mention of...

 

From time to time, I will start a new thread.  These new threads may get 100 views in a matter of a few minutes.

 

Certainly there are a lot of members at all times, looking & watching.  I suspect that there are also a goodly amount of guests.  I also like to think that people like to see what I have to say...

 

BUT

 

such a high number of views in such a short time leads me to think that there may be "bots" scanning this board.

 

Perhaps they are looking/watching for certain things.  Perhaps some really sharp investors have bots programmed to look for certain companies, ticker symbols, addresses, phrases and so on.

 

Anybody have any knowledge/information on this?

[Cigarbutt]

You are right to think that your posts are worth reading.

But I see that the Pentagon has finally acknowledged their program after the thread that you initiated on UFOs .

Perhaps, somebody is watching you.

 

Here's a link from an unbiased source?

https://www.nytimes.com/2017/12/16/us/politics/pentagon-program-ufo-harry-reid.html

 

[DTEJD1997]

You are right to think that your posts are worth reading.

But I see that the FBI has finally acknowledged their program after the thread that you initiated on UFOs .

Perhaps, somebody is watching you.

 

Here's a link from an unbiased source?

https://www.nytimes.com/2017/12/16/us/politics/pentagon-program-ufo-harry-reid.html

 

Yes, I am just so shocked on TWO levels about this thing:

 

A). I created a thread about this VERY subject just a week or two before it has come out.  Just lucky timing on my part.

 

B). I am in EVEN GREATER SHOCK that there is not more news/discussion about this!  These reports are not a bunch of "Hillbillies" being hassled by aliens & flying saucers in the trailer park.

 

A USA aircraft carrier sees something "odd" on it's radars.  It sends out two jet interceptors to investigate.  The pilots are incredulous at what they see.  They record part of the encounter on camera!  You've got qualified witnesses, you've got tape of something, you've got radar impressions of something.  This is a UFO sighting on MULTIPLE levels (radar, witness, video camera).

 

While this is certainly not conclusive proof of anything other than something odd, I would sure as hell be investigating it very thoroughly.

 

I could very well see it being "swamp gas" but for the radar signatures...That is one of the key things.

 

Of course, now would be the perfect time to release "news" about "flying saucers"...almost everybody is pre-occupied by "sex pervert maniacs" and the political circus, the economy, North Korea and on and on.  Great way to hide something in plain sight...

[JayGatsby]

Right now it shows "253 Guests, 31 Users (4 Hidden)" so getting 100 views in a few minutes doesn't sound impossible.

 

I'm always open to a good conspiracy though... BUY CMG, BUY CMG.... now we wait and see.  8)

[premfan]

No knowledge. The thesis seems sound. Since i have also experienced this. In a mostly zero sum game. The end game is less arbs. Less secrets. The payoff now seems to have shifted subtlety to being right.

[EliG]

Don't you people know about search bots? I thought it was a common knowledge even for non-computer types.

 

The search engines (Google, Bing, Yahoo et al) have bots that constantly crawl the web to index the new content, be it brand new pages or changes to the old pages. These bots may or may not count as views, depending on how this forum is configured.

 

You can ask Parsad to tweak forum configuration to show the bots as visible active users. If he chooses to do that, you would be able to see that search bots are present at all times.

 

 

 

[augustabound]

Don't you people know about search bots? I thought it was a common knowledge even for non-computer types.

 

Could you be more condescending?

[EliG]

I am sorry, I didn't mean to. I was genuinely surprised.

[augustabound]

I am sorry, I didn't mean to. I was genuinely surprised.

 

Fair enough, I jumped the gun a bit with my response. 

I'm studying software development for a 2nd career and never gave search bots a thought.

 

Next thing you're going to tell me is some company tracks our searches........  ;D

[rkbabang]

Don't you people know about search bots? I thought it was a common knowledge even for non-computer types.

 

The search engines (Google, Bing, Yahoo et al) have bots that constantly crawl the web to index the new content, be it brand new pages or changes to the old pages. These bots may or may not count as views, depending on how this forum is configured.

 

You can ask Parsad to tweak forum configuration to show the bots as visible active users. If he chooses to do that, you would be able to see that search bots are present at all times.

 

I know about search bots, but would that account for hundreds of views to a single page in minutes?  There is no need for google to load the page more than once (unless it changes).  I can't think of 100 search engines, are there that many?  Maybe there are a lot of other people/companies/government agencies/aliens who are crawling/indexing the web for whatever reason.  I don't know. 

 

[StubbleJumper]

Well, based on a couple of threads from a few years ago, I came to the understanding that many people access COBF posts through an RSS feed.  Is it possible that the reader-count goes up automatically when a post is delivered to an RSS feed even if the recipient never bothers to read the info?

 

 

SJ

[John Hjorth]

I think you are right about that assumption, StubbleJumper,

 

Google has created a special bot for RSS feeds, called Google Feedfetcher.

[EliG]

Don't you people know about search bots? I thought it was a common knowledge even for non-computer types.

 

The search engines (Google, Bing, Yahoo et al) have bots that constantly crawl the web to index the new content, be it brand new pages or changes to the old pages. These bots may or may not count as views, depending on how this forum is configured.

 

You can ask Parsad to tweak forum configuration to show the bots as visible active users. If he chooses to do that, you would be able to see that search bots are present at all times.

 

I know about search bots, but would that account for hundreds of views to a single page in minutes?  There is no need for google to load the page more than once (unless it changes).  I can't think of 100 search engines, are there that many?  Maybe there are a lot of other people/companies/government agencies/aliens who are crawling/indexing the web for whatever reason.  I don't know.

 

There are hundreds of known bots out there, plus many more unknown. Several web sites maintain the lists of known bots, for example:

 

http://www.user-agents.org/

http://www.robotstxt.org/db.html

https://github.com/monperrus/crawler-user-agents/

 

I have no idea which bots are active on COBF. An admin user should be able to figure that out.

[DTEJD1997]

Don't you people know about search bots? I thought it was a common knowledge even for non-computer types.

 

The search engines (Google, Bing, Yahoo et al) have bots that constantly crawl the web to index the new content, be it brand new pages or changes to the old pages. These bots may or may not count as views, depending on how this forum is configured.

 

You can ask Parsad to tweak forum configuration to show the bots as visible active users. If he chooses to do that, you would be able to see that search bots are present at all times.

 

Yes, I know about the Google & Yahoo!.  Obviously those are crawling/watching all the time...  I would also think that those would go without even saying....

 

As someone else pointed out, the general "indexing bots" should be fairly limited.  Give maybe a dozen hits to those?  Also, would those general indexing bots hit a new post within seconds/minutes?

 

The point I was trying to make is that I suspect that there are SPECIFIC bots that watch "The Corner".  We may not be their only target...but I would not be surprised if hedge funds/investment firms/investors are watching this forum (and others).

 

[John Hjorth]

Thank you for sharing your knowledge on this topic, Eli,

 

It's much appreciated.

 

Here is an article about robot.txt: Link.

 

Please try to google one of your own posts here on CoBF! As far as I can see, the GoogleBot does not index CoBF. I suppose Sanjeev has kicked it out, by the use of robot.txt in the root of installation on the netserver at the company hosting CoBF. Good for us. [<-Wrong! - Please see later post by EliG].

 

- - - o 0 o - - -

 

- Some time ago I got called out by fellow board members - for bad behavior - here on CoBF - with regard to Ms. Margrethe Vestager - while she was after one of the FAANGs. [Now she's "doing" IKEA.] I actually then did a google search for my post, and it didn't show up. Moreover, she hasen't sued my butt for slander - yet.

[writser]

The point I was trying to make is that I suspect that there are SPECIFIC bots that watch "The Corner".  We may not be their only target...but I would not be surprised if hedge funds/investment firms/investors are watching this forum (and others).

 

I will teach them a lesson by posting terrible ideas that will lose them a shitload of money! Wait .. I'm already doing that.

[John Hjorth]

You just made my day, writser! [ : - D ]

[EliG]

Here is an article about robot.txt: Link.

 

Please try to google one of your own posts here on CoBF! As far as I can see, the GoogleBot does not index CoBF. I suppose Sanjeev has kicked it out, by the use of robot.txt in the root of installation on the netserver at the company hosting CoBF. Good for us.

 

robots.txt is a public file:

 

http://www.cornerofberkshireandfairfax.ca/robots.txt

 

The file hides one directory (/wp-admin) from all bots. Aside from that, it doesn't block any bots.

 

For comparison, here's robots.txt from another forum:

 

http://www.financialwisdomforum.org/robots.txt

 

As you can see, it blocks a whole bunch of bots for bad behaviour, typically high resource consumption.

 

 

As for Google, it does index COBF, although with some delay. For example, I get 45 hits if I google "Hjorth BAM".

 

https://www.google.ca/search?q=site:cornerofberkshireandfairfax.ca/forum+Hjorth+BAM

 

The same search returns zero hits if I set the time range to the past 24 hours:

 

https://www.google.ca/search?q=site:cornerofberkshireandfairfax.ca/forum+Hjorth+BAM&tbs=qdr:d

 

I know that you posted in the BAM thread in the past 24 hours; lack of Google hits tells me that Google indexing is lagging. I don't know by how much.

[John Hjorth]

Thanks for setting this and me straight and teaching me here, Eli. Obviously the delay led me to wrong conclusion.

[DTEJD1997]

hey all:

 

I started a new thread today.  It was in the general category.  The topic was "grocery stores" and was started around lunch time on the East coast of the USA.  I would think that this is not a particularly exciting or inflammatory topic.

 

In 10 minutes there were about 70 hits.  3 of those were mine.

 

Obviously other members are looking, but I am sure there are automated bots watching too!

 

[premfan]

hey all:

 

I started a new thread today.  It was in the general category.  The topic was "grocery stores" and was started around lunch time on the East coast of the USA.  I would think that this is not a particularly exciting or inflammatory topic.

 

In 10 minutes there were about 70 hits.  3 of those were mine.

 

Obviously other members are looking, but I am sure there are automated bots watching too!

 

Interesting.. I believe its time for our chairman to do a paywall....

[rb]

This has been going on for a long time. I was getting lots of hits on searches to CoBF before I became a member. The posts were of such quality that I decided to fork over $20 to Parsad to become a member and stop mooching.

[compoundinglife]

I highly doubt there is a hedge fund scraping the site to pump into their algos but I guess anything is possible. What you are seeing is a fact of life on the Internet. Most traffic is not humans. The site could be updated/configured to filter out the bot traffic when calculating thread views. Paywall is not the answer, the internet and this website is more useful with bots than without them.

 

Some reports indicate that bots account for around 50% of web traffic. There are good bots and bad bots. Bots make our search work well, they make things like the Way Back Machine possible so you can see what sites used to look like years ago ( https://archive.org/web/ ), they do research on how the internet changes. Search engines do more than just index the data on the site. They regularly probe sites for availability, content changes, response time, security issues and more. They use this all this data in their search ranking. There are many many more search engines than yahoo and google.

 

I have written a bot in the past to scan the investment-ideas section of the site. I did it for shits and giggles to do some statistical analysis on the posts. I have only run it against the site once. But as long you do it slow and politely to not cause problems it is not a big deal.

[jouni1]

i wonder if these hedge fund bots are making the cannabis stocks pumped in investment ideas go up  ::)

[premfan]

I highly doubt there is a hedge fund scraping the site to pump into their algos but I guess anything is possible. What you are seeing is a fact of life on the Internet. Most traffic is not humans. The site could be updated/configured to filter out the bot traffic when calculating thread views. Paywall is not the answer, the internet and this website is more useful with bots than without them.

 

Some reports indicate that bots account for around 50% of web traffic. There are good bots and bad bots. Bots make our search work well, they make things like the Way Back Machine possible so you can see what sites used to look like years ago ( https://archive.org/web/ ), they do research on how the internet changes. Search engines do more than just index the data on the site. They regularly probe sites for availability, content changes, response time, security issues and more. They use this all this data in their search ranking. There are many many more search engines than yahoo and google.

 

I have written a bot in the past to scan the investment-ideas section of the site. I did it for shits and giggles to do some statistical analysis on the posts. I have only run it against the site once. But as long you do it slow and politely to not cause problems it is not a big deal.

 

Great post thank you. The dougishere entity that deleted his post is also on Reddit security analysis. He seems to behave differently on that site. Fascinating that platforms change behavior.