As the title says; the content that i'm getting is different than the actual content on the page. I'm trying to get a "href" info and a text inside of "p" tag(class=center-block) from a certain group of tags. When try to get "href" attribute from a link(second link below), it returns "/Login" text instead of what's inside of href attribute. And similarly when i try get the inner text of a "p" tag, it returns "Company Name" text. So it completely returns different values than actual contents. I could reach most of others but couldn't get the actual contents of these two elements specifically, even though i could see the actual contents when i inspect the elements in browsers.
Python code:
r = requests.get('https://www.examplesite.com/winners', headers=headers)
soup = BeautifulSoup(r.content, 'lxml')
list_of_winners = [item for item in soup.find_all('div', class_='bottom')]
list_of_winners.remove(list_of_winners[0]) # 0 indexed element is none, remove it from the list.
index_counter = 0 # index counter for dictionaries
for i in list_of_winners:
## Get the link
self.contest_link[index_counter] = i.find('a')['href']
## Get the title
self.contest_title[index_counter] = i.find('p', class_='center-block').text.strip()
index_counter += 1
index_counter = 0 # Reset the index counter
The html code:
<div class="bottom">
<div class="likes_count">
<a href="javascript:void()" style="color: #333;" class="contest-like-unlike" data-type="like" data-contest-id="24454">
<i class="fa fa-heart-o"></i>
</a>
<span class="popover_win">9 Likes</span>
</div>
<a style="float: right" href="https://www.example.com/retail-business/retail-business-24604">
<p class="center-block">Retail Business</p>
</a><span> </span>
</div>