they can do plenty enough to be worried. maybe they can not harm you physically (for now), but by having access to details of the private lives of people, their conversations, and being able to see how they form their opinions, they can use that information to determine how can they reshape public opinion on topics of their interests. this information can be used by themselves, or they can pass it to an ally, and it could be used to change almost anything, like interfere with elections, or further erode the need for privacy so that people are willingly giving up even more data to them