The oldest "open problems" are well known, duplication, trisection and quadrature were widely discussed in antiquity, the presumption is that by the time of Archimedes and Apollonius the experts realized that they are unsolvable with straightedge and compass (this is supported by Pappus's classification of problems into plane, solid and "mechanical"), but nobody could even frame an impossibility proof at the time, see Crippa's Impossibility Results on how this field emerged. Another old one is Dido's isoperimetric problem. Simplicius claims that circle was "known" to have the greatest area among figures with the same perimeter "by the time of Aristotle" and Archimedes and Zenodorus "proved it". The surviving fragments of Zenodorus show that he only dealt with the simplified problem for polygons, and it is unclear how the isoperimetric property could have been "proved" by Greeks' methods.
As for conjectures, they were not the business they are today. Confessing in writing one's inability to prove a suspected theorem was rarely done even in 19-th century, and when it was it was usually in a passing remark, not a Conjecture (as did Riemann with the zeta function zeros). In 17-18th centuries some conjectures were stated in private correspondence as "theorems", e.g. Goldbach in a letter to Euler so reports Fermat's contention that $2^{2^n}+1$ is always prime. More often, conjectures were communicated to students privately (compare to Arnold's intimation in Geometrical Methods of ODE:"unmotivated definitions that hide fundamental ideas and methods are like parables only explained privately to students") or inferred by others from peculiarities of the public text.
The prototypical example is Euclid avoiding the use of the parallel postulate in Elements for as long as he could, which was taken as a cue to prove it. In another instance, Apollonius boasted in the preface to Conics:"I observed that Euclid had not worked out the synthesis of the locus with respect to three and four lines, but only a chance portion of it and that not successfully: for it was not possible that the synthesis could have been completed without my additional discoveries".
An even earlier example is alluded to by Plato in Theaetetus, see McCabe's Theodorus' Irrationality Proofs:
"Theodorus here was drawing some figures for us in illustration of roots, showing that squares containing three square feet and five square feet are not commensurable in length with the unit of the foot, and so, selecting each one in its turn up to the square containing seventeen square feet; and at that he stopped."
Presumably, he did not stop because he thought that $\sqrt{19}$ was rational but because his even/odd method of proof faltered. The now familiar method from Euclid's Elements working for all primes was found by Theaetetus, his student, who presumably inherited the problem.
Archimedes wrote a famous letter to Eratosthenes, opening with "I formerly sent you some theorems discovered by myself with the request that you find the proofs which I provisionally withheld", where he reported a device for generating conjectures, his method of balancing weights. He expressed hope that others would find new theorems using it, but the only results he listed specifically were those already reproved by the Eudoxian double reductio. According to Toomer's interpretation of the preface to Diocles's On Burning Mirrors, Dositheus, Archimedes's friend and frequent correspondent, discovered the focal property of the parabola "by practical means", which Diocles undertakes to prove (he also discovered the spiral which Archimedes studied). This interpretation is controversial, however, see Acerbi's paper.
Euclid does not state anything remotely like conjectures about the perfect numbers. Nichomachus of Gerasa (c. 100 AD), a neo-Pythagorean, has a passage in his Arithmetic where he declares:
"Only one is found among the units, only one other among the tens, and a third in the rank of the hundreds, and a fourth within the limits of the thousands... And it is their accompanying characteristic to end alternately in 6 or 8, and always to be even".
Nichomachus does not qualify these as conjectures nor offers even a hint of arguments for them. The first was taken to mean that there is one perfect in each order of magnitude, in particular that there are infinitely many of them. Al-Bagdhadi (c. 980 AD) commented on them thus:
"He who affirms that there is only one perfect number in each power of 10 is wrong; there is no perfect number between ten thousand and one hundred thousand. He who affirms that all perfect numbers end with the figure 6 or 8 is right".
"He" would be right only if the odd perfect numbers do not exist, of course. Mersenne followed suit in 1644 by publishing a list of "Mersenne primes" up to 257. He did not bill it as a conjecture either, but a couple were extraneous and some missing. Still, he was remarkably close, see How did Mersenne discover Mersenne primes?
Stepping back a bit, here is from Schappacher's paper on Diophantus:
"Around 940 al-Khazin refuted an argument that abu-M. al-Khujandi
had proposed to show that the equation (in our notation) $x^3 + y^3 = z^3$ has
no solution in positive integers, and this discussion was carried on further
involving also Abdallah ben Ali! And it was also al-Khazin, and not Diophantus, who formulated the problem which has become over the past 15
years among arithmetic algebraic geometers a favourite topic of lectures for
a wider audience:
Congruent Number Problem. Decide whether a given squarefree integer is the area of a right triangle with rational sides. It is still unsolved, although we today have a very simple conjectural answer..."
Omar Khayyám, after classifying all cases of the cubic and solving each by intersecting conic sections, wrote in Al-jabr w'al-Muqabala:
"When however the object of the problem is an absolute number neither we nor any of those who are concerned with algebra have been able to solve this equation - perhaps others who follow us will be able to fill the gap..."
Others tried in vain, and in 1494 Summa Arithmeticae Luca Pacioli went so far as to say that algebraically "solving the cubic is as impossible as the quadrature". This was 19 years before del Ferro did it... and withheld the method from all but his student Fiore, see How was geometry historically used to solve polynomial equations?