The FDA's analysis says (pdf warning) :

Modafinil has not been evaluated in patients with a recent history of myocardial infarction or unstable angina, and such patients should be treated with caution.

In clinical studies of PROVIGIL, signs and symptoms including chest pain, palpitations, dyspnea, and transient ischemic T-wave changes on ECG were observed in three subjects in association with mitral valve prolapse or left ventricular hypertrophy. It is recommended that PROVIGIL tablets not be used in patients with a history of left ventricular hypertrophy or in patients with mitral valve prolapse who have experienced the mitral valve prolapse syndrome when previously receiving CNS stimulants. Such signs may include but are not limited to ischemic ECG changes, chest pain, or arrhythmia. If new onset of any of these symptoms occurs, consider cardiac evaluation.

Blood pressure monitoring in short-term (<3 months) controlled trials showed no clinically significant changes in mean systolic and diastolic blood pressure in patients receiving PROVIGIL as compared to placebo. However, a retrospective analysis of the use of antihypertensive medication in these studies showed that a greater proportion of patients on PROVIGIL required new or increased use of antihypertensive medications (2.4%) compared to patients on placebo (0.7%). The differential use was slightly larger when only studies in OSAHS were included, with 3.4% of patients on PROVIGIL and 1.1% of patients on placebo requiring such alterations in the use of antihypertensive medication. Increased monitoring of blood pressure may be appropriate in patients on PROVIGIL.

I'm not sure we have large enough numbers to give meaningful analysis. I could give a 25% (+/- 22) confidence that you'd likely experience elevated blood pressure, but you'd experience far greater harm from everyday stress or lack-of-exercise. There's a nice big list of serious adverse effects, but two or four out of seven hundred people doesn't seem very useful.

Dietary and psychiatric concerns may also be relevant.